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I TSTF.ATT? 

Dear NEAIR Colleagues, 

th 

The 39 Annual NEAIR Conference held in Bethesda, Maryland, Novemher 3-6, 2012 encouraged attendees to 
contrihute to our conference theme of Time for Decisions: Visualizing the Future. Three hundred and nine 
conference attendees had the opportunity to share and gain invaluable information from institutional research 
and higher education colleagues. The 2012 Conference Proceedings is a result of the conference theme in 
action. 

The Conference Program team led hy Program Chair Mark Palladino and Associate Program Chair Jessica Shedd 
developed a program filled with plenty of variety that included four plenary/keynote sessions, 1 1 contrihuted 
papers, 32 workshares, five techshares, 13 special interest groups, and four table topics. Poster Session Coordinator 
Paula Maas organized 16 posters to be on display. These offerings went through a blind peer review process 
facilitated by 74 proposal reviewers and coordinated by Mark Eckstein. Pre-Conference Workshop Coordinator 
Alan Sturtz organized 15 workshops with 166 participants. Exhibitor Coordinator Gurvinder Khaneja and Beth 
Simpson partnered with 18 exhibitors who offered 8 exhibitor showcases and Lightning Talks. 

A big thanks goes to Publications Coordinator Cristi Carson for all her hard work and keen eye editing the 
conference program, as well as compiling and organizing the 2012 Conference Proceedings. The 2012 Conference 
Proceedings contains papers submitted by authors as well as the 2012 Best Paper Award recipients. The award 
recipients were determined by Best Paper Chair Matthew Hendrickson and his committee. The 2012 Best Paper 
this year was awarded to Dehra Allen and Theodore Coladarci of the University of Maine for their paper. 
Examining the Threat of Nonresponse Bias as a Follow-up to the National Survey of Student Engagement. The 
2012 Best Eirst Paper was awarded to Chunmei Yao of SUNY College at Oneonta for the paper. Using Internal 
Market Ratios to Detect Gender Differences in Faculty Salaries. The 2012 Best IR/Practitioner Award was 
awarded to Laura Ariovich and William Richman of Prince George’s Community College for their workshare 
session. Non-returner Survey: Why Students Leave. 

Local Arrangements Chair Shannon Tinney Lichtinger and Local Arrangements Coordinators Arlene 
Blaylock, John Burczek Dreier, Elizabeth Clune-Kneuer, and Connie Pierson worked hard coordinating 
hotel, travel logistics and made sure we all enjoyed the NEAIR Third Place and all that Bethesda had to offer. 
Conference Website Coordinator Chris Choncek and Administrative Coordinator Beth Simpson developed and 
maintained the conference website as well as conference registration. Next year’s conference planning will be 
facilitated by online evaluations analyzed by Evaluation Coordinator Laura Uerling. 

It was a pleasure to work with such an extraordinary Conference Planning Team and the many talented volunteers. 
A premiere professional development opportunity was the result of the efforts of these individuals. We hope you 
take advantage of all the great information the 2012 Conference Proceedings have to offer! 

Best wishes. 


Stephen W. Thorpe, Ed.D 
NEAIR President 2011-2012 


INEAIR 

'■ North East Association for Institutional Research 


Leadership & Committees 

2011-2012 


Officers: 


Steering Committee Members: 


Administrative Coordinator (ex-officio) 


President 

Stephen Thorpe 

President-Elect 

Catherine Alvord 

Secretary (’ll-’ 14) 

Allison Walters 

Treasurer (’ 10-’ 14) 

George Rezendes 

Past President 

Gayle Link 

Program Chair 

Mark Palladino 

Local Arrangements Chair 

Shannon Lichtinger 

Technology Chair 

Chad May 

Member- At-Large (’09- ‘12) 

Emily Dibble 

Member-At-Large (’09-‘12) 

Paula Maas 

Member- At-Large (’ 10-‘ 13) 

Ann Marie Senior 

Member-At-Large (’ 10-‘ 13) 

Mary Goodhue Lynch 

Member-At-Large (’ 1 1-’ 14) 

Maren Hess 

Member-At-Large (’ 1 1-’ 14) 

Laura Uerling 

Ficio) 

Beth Simpson 


Standing Committees 


Local Arrangements (Standing Committee) 
Chair 

Exhibitor Coordinator 
AV Coordinator 
LAC Coordinators: 


Shannon Lichtinger 

Gurvinder Khaneja 
Jessica Shedd 
Arlene Baylock 
John Burczek Dreier 
Elizabeth Clune-Kneuer 
Connie Pierson 


Membership (Standing Committee) 

Chair Mary Goodhue Lynch 

Chair-elect Maren Hess 

Member Laura Uerling 

Member Betsy Carroll 

Member Jane Kimble 

Member Ingrid Skadberg 

Past Chair Paula Maas 


Prof Development Services (Standing Committee) 
Chair Cathy Alvord 

Member - Program Chair-Elect Alan StUltZ 
Member Emily Dibble 

Member Nancy Ludwig 

Member Laura Uerling 


Program (Standing Committee) 


Chair 

Associate Program Chair 
PCW Coordinator 
Evaluation Coordinator 
Newcomer/Mentor Coord 
Peer Review Coordinator 
Poster Session Coord. 
Publications Coordinator 


Mark Palladino 

Jessica Shedd 
Alan Sturtz 
Laura Uerling 
Mary Goodhue Lynch 
Mark Eckstein 
Paula Maas 
Cristi Carson 


Nominations (Standing Committee) 

Chair Gayle Fink 

Member - 2 YR Public Sector Gurvinder Khaneja 


Member 

Member 

Member 

Member 

Member 


Michelle Appel 
Becky Brodigan 
Elizabeth Clune-Kneuer 
Cherry Danielson 
Alexander Yin 


Site Selection (Standing Committee) 

Chair Cathy Alvord 

Treasurer George Rezendes 

Past LAC/Program Nicole Marano 

Admin Coord (ex officio) Beth Simpson 


Technology (Standing Committee) 

Chair Chad May 

Conf Web Coordinator Chris Choncek 
LAC (ex officio) Shannon Lichtinger 

Program Chair (ex officio) Mark Palladino 
Admin Coord, (ex officio) Beth Simpson 


Leadership & Committees 

2011-2012 



ad hoc Committees 

Finance Committee Grants Committee 


Chair 

Emily Dibble 

Chair 

Ann Marie Senior 

Treasurer 

George Rezendes 

Chair-Elect 

Laura Uerling 

Member 

Julie Alig 

Member 

Lisa Daniels 

Member 

Rebecca Brodigan 

Member 

Peter Feigenbaum 

Member 

Gayle Fink 

Member 

Jane Zeff 

Member 

Roland Pearsall 

Past Chair (ex officio) 

Michael Dooris 

Member 

Alan Sturtz 




Conference Proposal Peer Reviewers 



Mark Eckstein, Peer Review Coordinator 



Jacqueline 

Andrews 

Kristy 

Huntley 

Melissa 

Thorpe 

Louis 

Attinasi 

Elizabeth 

Johnston-O'Connor 

Steve 

Thorpe 

Rosanne 

Bangura 

Lisa 

Keating 

Rachel 

Tsang 

Melissa 

Barnett 

Ezekiel 

Kimball 

Laura 

Uerling 

R. 

Benson 

Daniel 

Larson 

Courtney 

Wade 

Gary 

Boden 

Melanie 

Larson 

Alex 

Wagner 

Ellen 

Boylan 

Ann 

Lehman 

Michael 

Whitcomb 

Christina 

Butler 

Carol 

Lerch 

William 

Wilson 

Jason 

Canales 

Paula 

Maas 

Charlotte 

Woodward 

Betsy 

Carroll 

Rajiv 

Malhotra 

Jasmine 

Yang 

Elizabeth 

Clune-Kneuer 

Tom 

McGuinness 

Alexander 

Yin 

Margaret 

Cohen 

Peggy 

McKernan 

Lillian 

Zhu 

Theodore 

Coladarci 

Pat 

Mizak 



Lauren 

Conoscenti 

Louise 

Murray 



Lisa 

Daniels 

Denise 

Nadasen 



Cherry 

Danielson 

Mitchell 

Nesler 



Mary Ann 

DeMario 

Doug 

Nutter 



Veena 

Dhankher 

Tiffany 

Parker 



Chrisanthy 

Diamantopoulos 

Kim 

Pavlick 



Kate 

Doria 

Ellen 

Peters 



Michael 

Duggan 

Maria 

Piteros 



Judy 

Dunham 

Paul 

Prewitt-Ereilino 



Jennifer 

Dunseath 

Heather 

Roscoe 



Nora 

Galambos 

Elizabeth 

Shaffer 



Hong 

Gao 

Judith 

Shaw 



Susan 

Gracia 

Layla 

Shumnok 



Jessica 

Greene 

Sri 

Sitharaman 



Phil 

Handwerk 

Jacquelyn 

Stirn 



Laura 

Harrington 

Susan 

Tammaro 



Braden 

Hosch 

Joy 

Tatusko 



Christopher 

Hourigan 

Danielle 

Taylor 




ACKNOWLEDGMENT 


Contained within these pagers of the NEAIR 39* Annual Conference Proceedings are eight 
contributed conference papers authored by twelve NEAIR colleagues. 

Additionally, you will find the contributed conference session and keynote presentations just 
few clicks away on the NEAIR website under the Proceedings tab. These pages are only 
accessible to signed in NEAIR members. 

Special thanks go out to Stephen Thorpe, Mark Palladino and Beth Simpson for their 
contributions, oversight and support with all aspects of publication responsibilities during the 
course of this past year. 

Cristi Carson 

2012-2013 NEAIR Publications Chair 
University of Southern Maine 



Table of Contents 


Can a Marketing Campaign Increase Response Rates to Online Course Evaluations? 

Kimberly Puhala 

Developing Community College Peer Institutions: Methods, Measures, & Issues. 

Joanna Musial-Demurat and Bruce Szelest 

Best Paper 

Examining the Threat of Nonresponse Bias as a Follow-up to the National Survey of 
Student Engagement. 

Debra Allen and Theodore Coladarci 

Maximizing the Utility of Alumni Feedback. 

Susan Tammaro 

NCES Projection Methodology: Institutional Research Applications. 

J. R. Robles 

Best IR/Practitioner Paper 
Non-returner Survey: Why Students Leave. 

Laura Ariovich and Allen Richman 

Testing Differences: Contrasts in Group Perceptions of Mission and Identity. 

Ellen Boylan and Kim Pavlick 

Best First Paper 

Using Internal Market Ratios to Detect Gender Differences in Faculty Salaries. 

Chunmei Yao 


CAN A MARKETING CAMPAIGN INCREASE RESPONSE RATES? 


I 


Can a Marketing Campaign Increase Response Rates to Online Course Evaluations? 
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Abstract 

Switching from paper- and-pencil administration of surveys to online surveys has been shown to 
result in a dramatic decrease in response rates. Eewer studies have examined response rates 
when switching from paper- and-pencil course evaluations to online course evaluations. This 
study examined the impact of a social marketing campaign on participation rates for online 
course evaluations. In addition, the impact of having computer access and the use of pre- 
notification postcards were also examined. The marketing campaign consisted of advertising the 
course evaluation web link on large screen TVs, posters, through online messaging, and via 
instructor announcements in class. Analysis of variance indicated that there were no differences 
in response rates between marketing and non-marketing groups. Access to computers and using 
a paper-and-pencil evaluation form did increase response rates when compared to online course 
evaluations administered in classrooms that were not equipped with computers. Discussion 
focused on the impact of online course evaluations on process and suggestions for creating 
strong incentives for students to complete online course evaluations. 
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Introduction 

Implementing process efficiencies is paramount to institutional researchers, whose offices are 
often under resourced. One area that is frequently under the jurisdiction of the Institutional 
Research Office and is highly administrative in nature, as well as time-consuming, is the 
administration of course evaluations. Efficiencies can be gained by switching to electronic 
course evaluations that are administered online, yet there is often the very real fear that survey 
response rates will drop significantly (Anderson, Cain, & Bird, 2005; Dommeyer, Baum, Hanna, 
& Chapman, 2004; Norris & Conn, 2005; Nulty, 2008). Prior research has investigated ways to 
increase response rates for surveys, in general, but few studies specifically address the 
conversion of course evaluations from in-class paper-and-pencil administration to online 
formats. 

At Quincy College, course evaluations are administered via paper-and-pencil scannable forms to 
every student in every class, for a total of approximately 14,000 course evaluation forms per 
semester. Problems with this approach include the use of already limited instruction time to 
complete course evaluations, the administrative burden of distributing course evaluation forms, 
and the staff time consumed by scanning forms and hand entering student comments. During the 
past summer, we implemented a pilot study of the use of online course evaluations. A link to an 
online survey was posted on the Quincy College portal, a password-protected course 
management system. Instructors were provided with memos asking them to announce the course 
evaluations in class, and containing screenshots of how to access the survey. The pilot study 
indicated that response rates dropped from 58.2% of students completing evaluations in Spring 
(administered in-class, paper-and-pencil) to 32.2% in Summer (announced in class, administered 
online). On the other hand, the percentage of classes with non-zero completion rates increased 
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from 83.2% to 88.1%, which led us to conclude that the issue is motivating students to complete 
the online course evaluations, rather than a breakdown in the communication of the new course 
evaluation process. 

A review of the literature indicates that response rates can be recovered for online course 
evaluations through the use of several methods. The literature indicates that offering extra credit 
or early access to course grades can return response rates to those found with paper- and-pencil 
in-class administration of course evaluations (Anderson, et ah, 2005; Dommeyer, et ah, 2004). 
However, there may be objections by faculty or administrators to use grade incentives on the 
grounds that this method is coercive or may bias study results (Dommeyer, et ah, 2004). 
Additional studies have examined the use of incentives other than grades to boost response rates. 
Strategies such as having faculty stress the importance of course evaluations, sending reminder 
emails to students, posting the survey link on the course calendar, ensuring that responses are 
anonymous, providing lottery incentives, and showing the students how to access the online 
survey have all shown to improve response rates to online course evaluations (Anderson, et ah, 
2005; Dommeyer, et ah, 2004; Norris & Conn, 2005; Nulty, 2008). In addition, Nulty (2008) 
indicates that using multiple methods further increases response rates. Kaplowitz, Hadlock, & 
Eevine found that the use of pre-notification postcards is effective at boosting response rates to 
online surveys (as cited in Wren & Showers, p.4). In addition, variations in response rates may 
be a function of logistics. Institutions with a higher number of computers per student have 
higher response rates to online surveys (Porter & Umbach, 2006). These last two areas have not 
been assessed specifically in regard to online course evaluations, which my research study 


proposes to do. 
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My research study will also introduce a novel approach to increasing response rates that draws 
on techniques used in other fields. One approach that has been utilized by the field of Public 
Health for decades is the use of media campaigns to initiate behavior change. This study will 
examine the use of a media campaign designed specifically around course evaluations. 

The main research question is whether or not a social marketing campaign can increase response 
rates to online course evaluations. This study will also examine the additive effect of two other 
techniques: the use of a pre-notification postcard and the accessibility of classroom computers. 
Pre-notification postcards will consist of small cards with the marketing tagline and the web link 
that will be distributed to students during class. All classes that teach in a computer lab will 
complete the online evaluations during class time. An analysis of variance will be used to 
examine the difference in response rates between the groups. The research hypotheses are as 
follows: 

1. The marketing campaign will increase course evaluation response rates compared to the 
overall online evaluation response rates (32.2%) observed in a prior semester. 

2. There will be higher response rates for those courses that are held in computer-equipped 
classrooms and exposed to the marketing campaign than those exposed to the marketing 
campaign alone. 

3. Response rates for courses held in computer-equipped classrooms should be equal to 
response rates (58.2%) seen in classrooms that complete paper-and-pencil course 
evaluations. 

4. There will be a higher response rate for those courses that receive pre-notification 
postcards than those exposed to the marketing campaign alone. 
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Methods 

All courses that ran at Quincy College in Eall, 2011 except independent studies or internships 
were included in the analysis, for a total of 718 courses. Courses were assigned to one of eight 
possible study groups, which differed based on type of course evaluation (paper- and-pencil 
versus online), whether the course finished before or after the start of a marketing campaign, 
whether or not the course was held in a computer-equipped classroom, and whether or not the 
course instructor handed out pre- notification postcards. Response rates were examined across 
the eight study conditions, and group differences in response rates were analyzed using an 
analysis of variance methodology. 

Course Evaluations 

The course evaluation form consisted of 46 questions. The first 26 questions asked the student to 
rate various statements about the course on a 5-point Eikert scale, from Strongly Disagree to 
Strongly Agree. There was also a choice to indicate that the statement did not apply to the 
course. Questions were organized in six categories, including Course Organization and 
Planning, Communication, Eaculty/Student Interaction, Assignments/Exams/Grading, 
Instructional Methods, and Student Engagement. Additional questions asked students to rate 
their satisfaction with different elements of the classroom. Student demographic information 
was also collected, and there were three open-ended questions. Questions on the online course 
evaluations were identical to the paper- and-pencil version, with the exception that students 
answered one additional question about whether they preferred to do course evaluations online or 
via the paper- and-pencil version. Online course evaluations were administered through 
surveymonkey.com, and students could access the link from any internet-enabled computer and 
were not required to complete the evaluations in class, except for the courses held in computer 
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classrooms. No password or identification code was required to complete the online course 
evaluations so that there were no barriers to accessing the course evaluations. Course evaluation 
questions are included in Appendix A. 

Course Assignment to Groups 

Courses were assigned to study groups using both convenience and random assignment. Eull- 
time faculty are required by contract to have the students in their courses complete course 
evaluations via paper- and-pencil, scannable forms. Therefore, all courses taught by full-time 
contract faculty were assigned to the paper-and-pencil group. Courses that that ended after the 
start of the marketing campaign were examined (Paper/Marketing) and compared to courses that 
ended prior to the start of the marketing campaign (Paper/No Marketing). All courses taught by 
adjunct faculty were assigned to complete online course evaluations, since there is no 
requirement in their contract to complete paper-and-pencil evaluations. Courses held in 
computer-equipped classrooms that ended after the start of the marketing campaign were 
assigned to the Online/Computer/Marketing group, while those that ended before the start of the 
marketing campaign were assigned to the Online/Computer/ No Marketing group. Courses that 
were not taught in a computer-equipped classroom and were taught by adjunct faculty were also 
assigned to online course evaluations. Those that ended prior to the start of the marketing 
campaign were assigned to the Online/No Marketing group, and the remaining courses were 
randomly assigned to either receive pre-notification postcards (Online/Postcard/Marketing) or to 
not receive pre-notification postcards (Online/Marketing). Courses that were held exclusively 
online were also included as a control group. The number of courses assigned to each study 
group and the number of students in each study group are presented in Table 1. 
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Table 1. Number of Courses and Students in Each Study Group 


Study Group 

# Courses 

# Students Enrolled 

Paper/Marketing 

170 

4,280 

Paper/No Marketing 

9 

224 

Online/Computer/Marketing 

41 

674 

Online/Computer/No Marketing 

23 

362 

Online/Postcard/Marketing 

188 

3,585 

Online/Marketing Only 

213 

3,975 

Online/No Marketing 

37 

610 

Online Course 

37 

589 

Total 

718 

14,525 


Marketing Campaign 

The Department of Institutional Research and Assessment collaborated with the Marketing 
Department to create a marketing campaign that lasted for a three- week period. The marketing 
campaign included the following components: 

• The creation of a catchy tagline encouraging completion of the course evaluations 

• Posting the course evaluation tagline and link on the College’s large screen TVs 

• Placement of laminated signs at all college student use computers with the tagline 
and the survey link 

• Placement of large, cardboard posters announcing the campaign in each of three 
academic buildings 

• Placement of a reminder “button” on the college website 

• Distribution of the survey link via student email 

• Two email reminders to complete the course evaluation 
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• Eacebook and Twitter reminders to complete the course evaluations 

• Eaculty encouragement of students to complete online course evaluations with an 
emphasis on their importance 

Pre-notification Postcards 

Pre-notification postcards were 2 Vi inches wide by 1 inch high. They contained a partial Quincy 
College seal and the word “Evaluate” on the front. The back of the cards had the complete web 
link for the survey, and a scannable QR code to allow students to access the survey from their 
cell phones. In addition, the tagline, “Make your voice count at QC” was written on the back of 
the card. Postcards had a black background with white writing. 
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Results 

The response rates to the paper- and-pencil administration in the semesters prior to switching to 
online course evaluations were 60.6% in Eall 2010 and 58.2% in Spring 2011. A pilot test of the 
switch to online course evaluations was conducted in Summer 2011, and overall response rates 
fell to 32.2%. The overall response rate to the Eall 201 1 online course evaluations was 36.2%. 
Analysis of Variance was conducted and indicated that courses where students completed paper- 
and-pencil evaluations had higher overall response rates than those who completed online 
evaluations in a computer-equipped classroom, which had higher response rates than courses 
whose students completed online evaluations (63.8% vs. 41.4% versus 22.6%, E=238, p<.001). 


Response Rates by Evaluation Type 

70 . 0 % 

60 . 0 % 

50 . 0 % 

40 . 0 % 

30 . 0 % 

20 . 0 % 

10 . 0 % 

0 . 0 % 

Paper-and-Pencil Online Evaluations, Online Evaluations, 
evaluations computer-equipped non-computer 

classroom classroom 



Analysis of Variance was used to further investigate the differences in response rates between 
the eight study groups, and specifically, whether there were any differences in response rates 
based upon the marketing campaign and the pre-notification postcards. The analysis of variance 
indicated that there were significant differences in response rates among the eight groups (E=71, 
p<.001). However, post-hoc tests revealed that there were no significant differences in response 
rates between the paper-and-pencil administration of course evaluations whether or not they were 
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exposed to the marketing campaign (p=.078). In addition, there were no significant differences 
between response rates for online evaluations that were completed in computer-equipped 
classrooms, whether or not they were exposed to the marketing campaign (p=.345). Last, an 
analysis of variance conducted on the four online evaluation groups that were completed outside 
the classroom indicated that there were no significant differences, even if they received a pre- 
notification postcard (p=.671). Results are presented in Eigure 2. 


Figure 2. Response Rates for Fall 2011 Online Course Evaluations by Study Group 



CAN A MARKETING CAMPAIGN INCREASE RESPONSE RATES? 


12 


Discussion 

Overall, study results indicated that there were no increases in response rates as a function of 
either the marketing campaign or the pre-notification postcards. The conclusion drawn is that 
students need a stronger incentive in order to motivate them to complete course evaluations. In 
some ways, this is not surprising as we are seeing lower and lower response rates to surveys 
across the college. This could be a result of students getting bombarded with surveys and other 
electronic announcements. The results of this study were used to support the purchase of a new 
online evaluation system that is integrated with our college portal. Students will have their 
access restricted from the portal’s other functions if they do not complete the course evaluations. 
We are hypothesizing that this will be a strong enough incentive for students to complete the 
course evaluations each semester. This new system will be implemented in time for the Spring, 
2013 course evaluation cycle. 

There was some evidence that response rates were trending higher for the paper- and-pencil 
administration of course evaluations in courses that had been exposed to the marketing 
campaign. One problem may have been that there were only 9 courses in the paper-and- 
pencil/no marketing group, versus 170 in the paper- and-pencil/marketing group. The reason for 
this discrepancy was because of the small number of courses that ended prior to the start of the 
marketing campaign, which was out of the experimenter’s control and based on the limited 
number of 10-week courses offered during the semester. Perhaps replicating the study with a 
higher number of courses in the paper- and-pencil/no marketing group would yield significant 
differences in response rates. 

A surprising finding was that response rates for online evaluations completed in computer- 
equipped classrooms were lower than response rates for paper- and-pencil administration. This 
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was surprising because the evaluations should have been administered in class in the exact same 
manner that the paper- and-pencil evaluations were conducted, and therefore, all students should 
have completed the course evaluation if they were in class that day. It is possible that since all 
full-time faculty had to complete paper-and-pencil evaluations, while adjunct faculty completed 
online course evaluations. It is possible that the higher response rates were a function of faculty 
status, with full-time faculty being more committed to completing evaluations since the 
evaluation process is clearly delineated in their union contract. Also, another possibility is that 
instructors encountered technological difficulties with the online evaluations that contributed to 
lower response rates. In fact, several instructors had reported that they had trouble accessing the 
online course evaluation system. 

One element of this study that is applicable to institutional researchers is the efficiency 
improvements in workload required of online course evaluations versus paper-and-pencil 
administration. In this particular Institutional Research office, which consists of one ETE worker 
and one part-time work study student for a school with a headcount of 4,674, processing time 
from when all evaluations were received in the office to when the final report was available to 
the college administrators was reduced from six months to three months. The major time saving 
occurred as a result of not having to scan paper-and-pencil evaluation forms or enter student 
comments. The process to clean the data, analyze the data, and write up the results was 
equivalent to the process for the paper-and-pencil evaluations. Additional time savings resulted 
from not having to compile packets of evaluation forms for each instructor; when switching to 
the online course evaluation system, one memo was distributed to all instructors that were 
completing online course evaluations that contained instructions and the web link where the 
survey could be found. In addition, monetary savings were achieved. The cost of a yearly 
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subscription to surveymonkey.com is approximately $300, compared to approximately $1,500 
spent in purchasing scannable evaluation forms. Plus, administering online course evaluations is 
a much “greener” solution than utilizing paper-and-pencil evaluation forms. In conclusion, 
although the study hypotheses were not supported, there were process improvements that 
resulted from this study which led to the decision to purchase a new system that should result in 
a greater incentive for students to complete course evaluations. 
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APPENDIX A: Course Evaluation Questions 

Questions 1-26 were rated on a 5-point Eikert scale (Strongly Agree, Agree, Neither Agree nor 
Disagree, Strongly Disagree) and students were also offered the choice Doesn’t Apply. 

A. Organization and Planning 

1. The instructor’s expectations were clear. 

2. The classes for this course were held for the allotted hours. 

3. The content covered was consistent with the course objectives as stated in the 
syllabus. 

4. The instructor displayed knowledge of the subject. 

5. The instructor summarized or emphasized important points in class. 

B. Communication 

6. The instructor’s presentations were clear and well organized. 

7. The instructor communicated clearly in English (or the language used in the 
course). 

8. The instructor used challenging questions or problems. 

9. The instructor inspired me to learn. 

C. Eaculty/Student Interaction 

10. The instructor was accessible outside of class. 

11. The instructor was helpful and responsive to students. 

12. The instructor created a learning environment that encouraged open discussion 

D. Assignments, Exams, and Grading 

13. The information given to students about how they would be graded was clear. 

14. Assignments were returned in a timely manner. 

15. The instructor’s feedback on assignments and exams in this course was effective 
in helping me learn. 

16. The work load for this course was appropriate. 

E. Instructional Methods 

17. The articles, videos, internet, etc. were effective in helping me learn. 

18. The out of class assignments were effective in helping me learn. 

19. The in-class assignments were effective in helping me learn. 

20. The text(s) used in this course were effective in helping me learn. 

21. The instructor used a variety of teaching methods. 

E. Student Engagement 

22. This course contributed to my knowledge, skills or personal development. 

23. This course helped me to think independently about the subject matter. 

24. 1 studied and put effort into the course. 

25. 1 was challenged by this course. 

26. 1 would recommend this course to another student. 

G. Overall 

27. The pace of this course was: (Very East) (Somewhat East) (About right) 

(Somewhat Slow) (Very Slow) 

28. The overall quality of this course was: (Excellent) (Very Good) (Good) 

(Eair) (Poor) 
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Questions 29-35 were rated on a 5-point Eikert scale (Very Satisfied, Satisfied, Somewhat 
Satisfied, Not Very Satisfied, Not At All Satisfied) and students were also offered the choice 
Doesn’t Apply. 


H. Please rate your satisfaction with the classroom in each of the following: 

29. Size of the classroom 

30. Outside classroom noise/distractions 

31. Classroom temperature 

32. Classroom lighting 

33. Cleanliness of classroom 

34. Working order of Equipment and Technology 

35. Ease of use of the Quincy College Portal 

I. Student Information 

36. Are you currently in a Certificate or Associate Degree program at Quincy 
College? 

37. If in a certificate or degree program at Quincy College, what is your program of 
study/major? 

38. How many credits are you taking this semester? 

39. How many credits have you completed at Quincy College prior to this semester? 

40. What was the most important reason for your taking this class? (choose one): 
College Requirement, Elective, Because of the Instructor, Related to my work. 
Personal Interest 

41. Did you take the pre-requisite for this course? 

42. Do you communicate better in English or in another language? 

43. What is your gender? 

J. Comments 

44. What did you like about this course? 

45. What didn’t you like about this course? 

46. What aspects of the course would you change? 
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Introduction 


Community colleges serve almost half of the undergraduate students in the United 
States, providing open access to postsecondary education, preparing students for transfer to 
four-year institutions, and offering workforce development and skills training. Given the scope 
and importance of the community college role, it is difficult to overstate the importance of 
assessing and strengthening the quality of its education, performance and outcomes. Proper 
selection of peer groups is a key starting point in effective use of benchmarking for 
improvements and changes which can help to transform institutions to enhance their standing. 

Both external and internal forces create a need for development of accurate peer 
groups for community colleges (Bers, 2006). The external factors include pressure for 
accountability and efficiency and thus proof that colleges provide quality education, allocate 
appropriately their funds, and are effective in fulfilling their missions. Community colleges must 
show student attainment and institutional effectiveness through analyzing intra-institutional 
trends and inter-institutional benchmarking (Juhnke, 2006). The internal motivations of 
selecting peer groups include self-evaluations that may help to identify areas in which a college 
needs improvements and in which it is strong and needs to sustain its position. In other words, 
benchmarking can be used to focus college attention on practices, programs, and policies that 
may be in need of improvement - and on those worthy of celebration (McClenney, 2006). Both 
external and internal motivations support benchmarks and benchmarking (Bers, 2006). 

The Columbia-Greene Community College (CGCC) is one of 64 campuses in the State 
University of New York system. The system includes four university centers, nine other 
doctoral-granting institutions, thirteen university colleges, nine technology colleges, and twenty 
nine community colleges. The 29 community colleges are very diverse and so it is not obvious 
how any one of them can know which other community colleges would be appropriate peers 
for benchmarking. Our research aims at helping individual community colleges decide how to 
narrow their choices of peers in order to create comprehensive benchmarking groups. 

Columbia-Greene Community College, like other community colleges, has a need to 
develop a list of peer institutions to support its institutional planning and effectiveness, 
decision making processes, and planned initiatives. Its original peer list was successfully 
developed about fifteen years ago and has been updated a few times since then. The college 
primarily used informal panel reviews and a threshold method to create its original list of peers, 
and later during the updating process. The latest version of Columbia-Greene's peers is now 
some 8 years old. Although the list is still basically appropriate, it needs re-evaluation due to 
the fact that the last ten years have brought many changes in higher education, particularly 
within the SUNY system. Within New York State, and nationally, many colleges continually 
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adjust and transform themselves to meet new challenges and enhance their standing. These 
processes are visible across all institutions starting from research universities through university 
colleges and technical colleges, and community colleges. Some changes are commonly visible 
across groups of institutions; for example, residential student housing projects brought changes 
and new challenges to many community colleges in the last decade. To facilitate changes which 
occurred at Columbia-Greene Community College as well as those at our historical peers, we 
have decided to re-evaluate and revisit our peer group selection though applying an advanced 
method of peer selection used successfully for the same purposes by the University at Albany in 
1996 and 2011 (Szelest 1996; Musial-Demurat & Szelest, 2011). 

Literature Review 

Types of comparison groups 

Previous research has identified several peer types that have been successfully used 
during peer selections for four-year colleges and major research universities. Brinkman and 
Teeter (1992) listed the best known types of comparison groups; 1) competitors, which regard 
to applicants, faculty, or financial resources, 2) aspirational, those institutions we strive to be 
like in some respects, 3) predetermined, those institutions that are natural, traditional, or 
which share a common jurisdictional area, and 4) peers, which can be used in benchmarking. 
This classification of comparison groups, for the most part, remained constant across time. It 
helps with identification of a pool of institutions that should be selected based on the given 
situation and purpose of the comparison. 

Three distinctive comparison groups were identified for community colleges in the five 
benchmarking projects discussed by Hurley (2002a): peer institutions and peer groups, 
comparator institutions, and benchmarking institutions. Very similar peer types are used in 
newer publications on comparison groups for community colleges. Horn (2008) refers to "peer 
grouping" while describing the use of benchmarking in term of its development, mechanics, 
and implications for California's system of 109 community colleges. Juhnke (2006) uses the 
"peer institutions" term while discussing the National Community College Benchmark Project 
(NCCBP), which provides community colleges within a system to report data on key learning 
outcomes and indicators of institutional effectiveness and to compare their results with data 
from selected peer insinuations. Similarly, Manning and Bostian (2006) refer to "peer 
institutions" while discussing how Central Piedmont Community College (CPCC) has used data 
from the NCCBP to initiate and implement strategies that were demonstrated to be effective in 
reducing course-withdrawal rates. Another term -"community colleges clusters"- is used by 
Hurley (2009) in his study of the Community College Survey of Student Engagement (CCSSE) 
benchmarks for extra-large community colleges. 
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Methods 


There is no one generally accepted standard technique with which to identify peer 
institutions. Brinkman and Teeter (1987) define cluster analysis, hybrid, threshold, and panel 
reviews as useful methods for peer institution selection. These methods range from a purely 
statistical technique like cluster analysis to mainly subjective methods like a panel review. All of 
them have been used for development of peer institutions for four -year colleges and research 
universities. These institutions have much longer histories and stronger foundations in peer 
selections than community colleges do (Hurley, 2002b), so that community colleges have a 
significant need for benchmarks of education quality (McClenney, 2006). 

In recent years, several scholars and practitioners have focused on using cluster analysis 
for selecting peer groups or clusters for community colleges. For example. Hurley (2002) 
discusses how cluster analysis has been employed in various community college peer grouping 
efforts. This statistical analysis is also applied by Hurley (2009) in research on the formation of 
community college clusters using the CCSSE benchmarks for extra-large community colleges. 
Similarly, cluster analysis was used for peer grouping for California's system of 109 community 
colleges (Horn, 2008). One of the reasons that this statistical tool has been used for peer 
grouping on system-level benchmarking is that it prevents politically biased peer selections that 
would favor certain colleges. 

Hybrid approach 

Although, there are advantages of using cluster analysis for peer selections, especially 
for system-level analysis, in recent years several scholars have strongly recommended a hybrid 
approach as a useful tool in forming peer groups because it incorporates the advantages of 
both the data-driven analysis and expert judgment (Ingram 1995; Zhao 1997; Lang 2000; Xu 
2008; Archer, Reiss, Armacost, Sun, Fu 2009; Musial-Demurat, Szelest 2011). The approach is 
used for analyzing peer groups mostly for four-year colleges and its applicability to community 
college settings was not explored in detail. Thus our research aims to evaluate how the hybrid 
approach can be used for selection of institutional peers by community colleges. The research 
uses a combination of statistically driven techniques and subjective judgment methods in 
regard to selecting possible sets of peers, variable selection and weighting schemes. Overall, 
the structure of the research is based on methods and findings presented in the paper "In 
Search of Peer Institutions: Two Methods of Exploring and Determining Peer Institutions" 
(Szelest, 1996) and further developed peer selection ideas presented in Musial-Demurat & 
Szelest 2011. These two publications are thus the core starting points for the present research 
in terms of methodological approach. Additionally, however, we now focus on a different set of 
institutions. Whereas the prior methodological work was developed and tested for selection of 
peer four-year public research universities, this new research focuses on community colleges. 
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Dimensions 


Two-year colleges differ significantly from one another, and there is a dramatic variation 
in terms of institutional characteristics like size, geographic location, available resources, 
programs, and in terms of student characteristics like enrollment patterns. These differences 
must be taken into consideration in the benchmarking process, both when interpreting an 
individual institution's benchmark scores and when making institutional comparisons 
(McClenney, 2006). Dimensions like size, finance, quality, and complexity have been commonly 
used for peer analysis for four-year colleges and research universities (Szelest, 1996; Zhao 1997; 
Weeks, Puckett, Daron 2000; Xu 2008; Gaylor 2009; Nzeukou and Muntal 2010). Hurley (2002) 
recommends using these dimensions for identification of peer institutions for community 
colleges based on his evaluation of peer analyses conducted on the state and institutional 
levels. However, the validity and reliability associated with these institutional variables in the 
four-year college and universities levels have been established over last two decades. But this is 
new research focusing on community colleges (Hurley, 2002b). Thus there is a need for 
reevaluation of institutional dimensions/variables to determine which can best be used for peer 
selections for the two-year college setting, taking into consideration the unique symbiosis of 
community colleges and their communities. Factors like the social-economic profile of the 
surrounding community or wealth of the community are examples of factors that may be more 
critical for establishing benchmark groups for community colleges within the same state, as well 
as in other states. 

Variables 

Once important dimensions have been chosen, variables can be identified for 
determining peer institution selection. According to Aldenderfer and Blashfield (1984), the 
selection of variables in peer analyses is one of the most critical steps in the research process. 
That is because the choices of variables for peer comparisons strongly influence the 
characteristics of institutions and thereby the selection of peer groups. 

In many peer selection studies variables are placed into major categories, which 
frequently are similar or the same as dimensions. Interestingly, comparison of dimensions used 
by researchers for analysis peers for four-year instructions including research universities 
(Szelest, 1996) and two-year institutions (Hurley, 2002) shows that the same taxonomy- 
including institutional level, size, complexity, quality and finance-can be used for selecting 
peers for those institutions. However, there are some differences in terms of variables used for 
peer selections for four-year colleges and research universities and two-year community 
colleges. For example, variables used for measuring quality differ based on type of institution; 
SAT scores are frequently used in four-year colleges whereas most community colleges have an 
open enrollment policy and accept students without requiring these scores. Similarly, funding 
sources differ between different types of higher education institutions, with research 
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universities on one end of the spectrum paying a lot of attention to research expenditures and 
on the other end community colleges, which usually do not generate research funds. 

There is no consensus among scholars on the correct number of variables to use for 
peer comparison analyzes. Some researchers like Nzeukou and Muntal (2010) use a large 
number of variables grouped in a few dimensions in their analysis aiming to select peer 
institutions. Similarly, the ID Northwest peer institution report discuses many variables that are 
used during a peer selection process. Other researchers like Archer, Reiss, Armacost, Sun, Fu 
(2009) or Weeks, Puckett, Daron (2000) focus on fewer variables while selecting peer groups. 

Weights 

After making decisions on the type of comparison group, methods of selecting 
institutions, and dimensions and variables for peer analyses, it is time for another very 
important step: the choice of weighting scheme. There is no agreement among scholars 
whether weighting in necessary during the selection of peer institutions. Some scholars 
(Szelest, 1996; Xu, 2008) have made conscious decisions not to weight the variables/factors in 
their studies. One given reason rests on the assumption that variables/ factors may receive 
special considerations in pre-final analysis procedures so they do not need weighting in the final 
stage of peer comparison. For example, size variable is frequently considered in the initial 
processes of peer comparisons and is captured through multiple measures, so it may not need 
additional weight in the final step. Other researchers, such as Week et al (2000), have made a 
decision to weight the selection variables to give greater or lesser emphasis to key factors 
related to each campus's mission and programs. Similarly, Horn (2008) indicate that for peer 
grouping study for California's system of 109 community colleges staff needed to weight the 
specific criteria that applied to accountability programs. 

Summary 

Thus this present research not only surveys fifteen years (1996-2011) of scholarly 
literature on selecting peer institutions for four-year colleges including Szelest's (1996) 
methodology but also evaluates research on peer institutions for community colleges and 
explores how to adapt the methodology to the 2-year setting. As such, it highlights best 
practices around the most frequently used methodological approaches, dimensions upon which 
comparisons are often based, and the individual variables and metrics used in identifying peers. 
The research refers to approaches already employed (by others) on community colleges for 
peer selections such as usage of CCSSE benchmarks for formation of community college clusters 
(Hurley, 2009) or peer groupings implemented in California system of 109 community colleges 
(Horn, 2008). 

Dataset Development 

A dataset was developed to include 35 (public) NYS community colleges for potential 
analysis. Because the target institution to which peer selections are to be made is Columbia- 
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Greene Community College, a Carnegie Associate's--Public Rural-serving Small College (2010), 
potential peer institutions with the similar classification 'Associate's Public College' were 
chosen for exploration. Following the input from college leaders and the literature on 
intra/inter-state benchmarking, a decision was made early in the process to consider as 
possible peers only community colleges within New York State. The main reason for focusing on 
institutions from one state lies in fact that states vary considerably in the ways community 
colleges are regulated, governed, and funded. For example, in some states local taxes comprise 
a substantial revenue source for two year colleges, whereas in other states all funds are 
received from the state (Bers, 2006). Similarly, some states have statewide faculty contracts, 
which limit an institution's autonomy to make decisions about employees' salaries and benefits 
whereas other states have much open policies related to faculty employment (Bers, 2006). Thus 
a decision was made to limit the pool of possible peers to only New York State colleges taking in 
to consideration that Columbia-Greene Community College is located in this state. Additionally, 
the comparison group is restricted to public institutions due to the public control of the target 
institution, which is a part of the State University of New York (SUNY) system. Extraction of 
institutions with these criteria (public. New York State institution. Associate's College) from the 
IPEDS data analysis tool resulted in a dataset of 35 institutions comprising 29 community 
colleges from the SUNY system and 6 community colleges from the CUNY (City University of 
New York) system. 

Missing data are a concern in any quantitative analysis. This is true even with IPEDS 
data for which NCES has the power to impose monetary (and worse!) penalties for institutions 
that do not submit timely or accurate data. There are no doubt myriad reasons why data may 
show up as missing in IPEDS, including applicability, reporting errors, or even a result of 
extracting data from one particular IPEDS survey like the GASB financials survey as it appears 
that public universities should use that specific reporting form, but in reality a handful of public 
institutions use the FASB financial instrument. As discussed below, the analytic methods used in 
this research require complete data. Missing data excludes institutions from the analyses. 

These instances were very few, as only a few isolated instances of missing data were identified. 
In these instances, mean substitution was used. In addition, a flag was added to the underlying 
data file so that if campus leaders chose to inspect the raw data for any particular school, they 
could identify instances where mean substitution was used. 

The rationale for using mean substitution is that because of the number of measures 
used (43), it is believed that using the mean value on one or two measures will not adversely 
affect the analyses. Again, we included a flag in the dataset to mark those measures (and 
institutions) in which mean substitution was used so that we could revisit the details of the data 
should these institutions find their way into the final peer listing. Finally with respect to missing 
data, it is worth noting that there were no instances in which we needed to remove a 
prospective institution from the analysis. 
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Variables Development 

After selecting only public New York State Associate's Colleges, we chose variables 
based on their appropriateness in capturing dimensions of finance, size, complexity, and 
quality. We based our selection of variables on a re-evaluation of variables used in Szelest's 
study (1996) and examination of the most popular variables used by other researchers in the 
last fifteen years. Additionally, the top administrators were asked to identify which important 
variables they believe should be taken into consideration based on their experiences with 
selecting Columbia-Greene peers over the past fifteen years. That administrative input was 
essential for our selection of variables which become have become currently important for 
Columbia-Greene. The final analysis includes nine financial, five size, three quality, and twenty- 
six complexity measures. 

Finance 

The measures of institutional finance address both overall support and more specific 
expenditure functionality. We capture total from the IPEDS dataset. Specific expenditure 
account categories of instruction, academic support, student services, and institutional support 
are evaluated in terms of their share of core expenditures, as defined by IPEDS. Thus each 
specific expenditure is divided by the total operating and non-operating expenditures. Similarly, 
for calculation of total revenues we include total revenues from operating and non-operating 
revenues. In terms of percent of budget from different sources we evaluate the percentages of 
revenue generated from state appropriations, tuition and fees, and local appropriations. Local 
appropriations are of particular import as they constitute a unique revenue stream for the 
community college sector. 

Size 


The size measures are categorized in two groups of measuring: that represent the size 
of student body and the size of the faculty body. Total student FTE is used to capture both size 
and mission of institutions. FTE students are calculated by combining full-time headcount and 
one-third of the part-time headcount enrolment, per the IPEDS definition. Additionally, the size 
of the entering full-time first -time freshmen cohort as a percentage of all undergraduate 
students and entering transfer students as a percent of new students are included in the 
analyses. In terms of faculty body we evaluate the number of full-time and part-time 
employees, and the number of full-time tenured track and full-time instructional lecturers, 
which helps to illustrate some important employment patterns of community colleges. All size 
data come from IPEDS. 
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Complexity 


The twenty-six complexity measures address student body composition, faculty 
composition, technical emphasis, socio-economic characteristics of service counties, degree 
awarded distributions, and the residential nature of the campus. Commonly accepted 
indicators of student body complexity include students' socio-economic distribution based on 
the percent of freshmen who are Pell recipients, the percent of freshmen local grant recipients, 
and the percentage of institutional grant recipients. Additionally, age distribution as a 
proportion of undergraduates of traditional age (i.e., 18 to 24 year olds and 25 to 65 year olds), 
the percent of part-time students, and diversification of the student body defined by the 
percentage with minority classification are included in the analyses. To further evaluate the 
student body of community colleges we made a decision to evaluate the percentage of non- 
degree enrollment to reflect an essential component of the two-year college mission. We use 
one individual measurement of faculty complexity: the percent of full-time faculty who are 
tenured or tenure-track, which is a crucial indicator of composition of faculty body at 
community colleges. Emphasis on technical education derives from the percent of degrees 
awarded in technical majors versus the percent of degrees awarded in liberal art, social 
services, and business majors. Three indicators of social-economic profile of the surrounding 
community or wealth of the community are also incorporated into the analyses. These include: 
county(s) population (2011) served by a college, estimated median household income 2006- 
2010 from county(s) served by a college, and estimate unemployment rate (August 2011- 
September 2012) in a county(s). These variables are especially important when we take into 
consideration the unique symbiosis of community colleges and their communities. The 
residential nature is measured by the presence or absence of residential housing for students. 
Interestingly, a few years ago this indicator would not be relevant to community colleges, which 
in majority did not have residence halls. But more recently, many of them have made a 
decision to add on-campus housing options to their service offerings. Lastly, due to the fact that 
athletes are more likely to live in residence halls, we decided to include a measure for whether 
colleges are members of the National Junior College Athletic Association (NJCAA) or not. 

Quality 

Unfortunately, it is not easy to directly verify the quality of higher education institutions 
due to a lack of well-defined and measureable indicators. In particular, the traditional proxies 
for quality input variables are not available for community colleges which in majority have an 
open enrollment policy and enroll students with minimal requirements like a high school 
diploma or its' equivalent. In terms of quality output variables we focus our analysis on three 
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measurements: first year retention rate of full-time first-time freshmen, the graduation rate, 
and transfer out rate. The evaluation of these variable helps to illustrate some aspects of a 
college mission. The transfer out rate is an especially relevant indicator for community 
colleges, which in majority, prepare students not only for graduation but also for transfer to 
four-year colleges. 

Methodology 

As in previous peer institution explorations, we conducted a number of preliminary 
examinations by looking at institutional attributes such as athletics conference, the presence or 
absence residence halls, as well as by various demographic and ecological characteristics. In the 
past, we have found that surveying the institutional landscape and mapping out institutional 
typologies to be an excellent means of educating campus stakeholders about institutional 
similarities and dissimilarities. 


10 



Quintile Groupings of Local Support by Degree of Urbaneness 
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In addition to showing where each institution falls on these distributions, we have 
highlighted our City University of New York (CUNY) colleges in red. CUNY schools have separate 
funding and administrative funding structures within New York State, so it is important to 
highlight this difference for campus leaders. Other typologies developed for initial information 
sharing might include juxtaposing enrollment by the percentage of students receiving Pell 
grants, or student selectivity with other financial resource measures like percentage of budget 
allocated to instruction or student support services. 

Examining these institutional typologies is also informative for the analysts. Deans and 
senior staff often have "insider insights" into other institutions and their cultures owing to 
either personal experience or professional collaborations/interactions with their counterparts 
across the country. This can help inform variable and measure selection, as well as aid in adding 
additional context to the eventual result set. 

Once the institutional landscape is satisfactorily explored and campus leaders are 
familiarized with the data typologies, we then turn to statistical techniques to begin selecting 
peer institutions. The rank distance method used by Berthold (1996) is very similar to that used 
by the University of Kansas and described by Teeter and Christal (1987), but uses a percentile 
rank order of institutions on each measure rather than Z scores to calculate 
similarity/dissimilarity. A second method, cluster analysis, utilizes principal components analysis 
and factor scores to group universities that are determined to be similar across specified 
dimensions into clusters. This technique was developed by Terenzini et al. (1980), who were 
amongst this technique's early pioneers in the late 1970s. 

The Kansas classification described by Teeter and Christal (1987) utilized a weighting 
scheme to elevate the importance of certain variables after standardization. While the rank- 
distance analysis conducted for this research does not use explicit weights for the variables 
chosen, an implicit weighting scheme is active in that nine financial, five sizes, twenty-six 
complexity, and three quality measures are used in the analysis. Hence, elements of finance 
and size are more dominant in assessing institutional similarity/dissimilarity with the target 
institution. In addition, it should be noted that many of the measures used are highly correlated 
with each other, so they in effect may very well be measuring the same variable or dimension. 
Reducing the number of measures used in the analysis, and which measures, could influence 
where institutions fall out in proximity to the target institution. Hence our multi-method 
approach. 
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Due to space and visual presentation limitations, Table 1 shows only three of the nine Financials 
measures used; two of the five size measures, six of the twenty-six complexity measures, and 
three of the quality measures used. 

Table 1 - Initial Extraction using the Rank-Distance method 
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26 

2 

27 

27 

21 

34 

9 

19.5 

6 

34.5 

SUNY Cayuga CCC 

353 

20 

21 

24 

26 

31 

11 

5.5 

5 

33 

17 

20 

19.5 

20 

32 

SUNY Corning CC 

367 

24 

25 

7.5 


24 

22 

5 

24 

15 

32 

9 

26 

23.5 

16.5 

24.5 

SUNY Clinton CCC 

381 

34 

34 

17.5 


32 

29 

21 

34 

10.5 

30 

14 

22 

33 

12.5 

13 

SUNY Finger Lakes CC 

408 

22 

17 

32 


20 

17 

12 

24 

20 

28 

28 

17 

28.5 

3.5 

18 

SUNY Sullivan CCC 

431 

31 

32 

6 


35 

32 

27 

27 

4 

11 

8 

21 

35 

32 

1 

SUNY Schenectady CCC 

436 

27 

29 

35 


22 

27.5 

1 

11.5 

15 

20 

23 

12 

26 

26.5 

13 

SUNY Orange CC 

439 

16 

18 

4.5 


18 

15 

9 

24 

32.5 

10 

11 

6 

7.5 

26.5 

10 

SUNY North Country CC 

450 

35 

35 

20 


33 

35 

3 

17 

17.5 

34 

35 

28 

34 

20 

27 

SUNYTompkins Cortland ( 

456 

21 

23 

13.5 

23 

23 

7 

35 

15 

23 

31 

19 

21.5 

12.5 

2.5 

SUNY Mohawk Valley CC 

463 

18 

14 

28 


16 

19 

30 

11.5 

9 

19 

5 

24 

13.5 

16.5 

24.5 

SUNY Jamestown 

480 

26 

24 

17.5 


25 

24 

34 

1.5 

10.5 

27 

32 

32 

13.5 

1 

32 

SUNY Broome CCC 

490 

19 

19 

10.5 


17 

13 

35 

17 

25 

31 

2 

27 

16.5 

7 

34.5 

SUNY Genesee CC 

493 

23 

22 

32 


19 

30 

4 

30.5 

12.5 

29 

33 

18 

28.5 

5 

2.5 

SUNY Niagara CC 

496 

17 

20 

24 


15 

20 

25 

30.5 

29 

22 

21 

25 

11 

12.5 

29 

SUNY Rockland CC 

497 

15 

16 

3 


14 

18 

24 

10 

35 

9 

25 

3 

1 

20 

18 

CUNYHostos CC 

502 

12 

12 

1.5 


21 

14 

15 

17 

2 

2 

29 

34.5 

11 

33 

32 

SUNY Herkimer CC 

503 

25 

27 

28 


28 

27.5 

33 

30.5 

7.5 

16 

30 

33 

31.5 

3.5 

5 

SUNY ErieCC 

510 

9 

10 

28 


7 

8 

32 

3.5 

19 

14 

3 

23 

21.5 

23 

18 

CUNY Bronx CC 

526 

10 

7 

7.5 

12 

9 

13 

17 

1 

1 

7 

34.5 

16.5 

35 

29 

SUNY Dutchess CC 

529 

14 

15 

22 

13 

16 

6 

30.5 

34 

13 

18 

5 

9 

16.5 

18 

SUNY Onondaga CC 

531 

13 

13 

32 


11 

11 

17 

8.5 

23.5 

18 

4 

16 

25 

26.5 

13 

SUNY Westchester CC 

576 

7 

9 

9 


10 

12 

8 

1.5 

31 

7 

19 

4 

16.5 

34 

5 

CUNY LaGuardia CC 

585 

4 

4 

13.5 


6 

7 

14 

17 

6 

4 

13 

10.5 

4.5 

30 

29 

SUNY Hudson Valley CC 

588 

11 

11 

20 


9 

10 

16 

22 

28 

17 

1 

8 

28.5 

12.5 

5 

CUNT Queensborough CC 

617 

8 

8 

13.5 


8 

4 

23 

17 

21 

5 

6 

10.5 

2.5 

26.5 

7.5 

SUNY Suffolk CCC 

629 

2 

2 

13.5 


1 

3 

20 

27 

30 

15 

16 

2 

7.5 

26.5 

18 

CUNY Kingsborough CC 

639 

5 

6 

16 


5 

6 

22 

17 

7.5 

6 

24 

29 

4.5 

9 

24.5 

SUNY Monroe CC 

650 

6 

5 

34 


4 

5 

26 

3.5 

22 

12 

10 

14 

11 

16.5 

10 

CUNY Bourough Manhatta 

651 

3 

1 

28 

3 

2 

28 

17 

3 

3 

27 

7 

6 

31 

18 

SUNY Nassau CC 

700 

1 

3 

1.5 

2 

1 

31 

33 

32.5 

8 

12 

1 

2.5 

26.5 

7.5 


In order to explore the possible impact of collinearity among measures, as well as 
redundancy and attendant issues around inherent weighting by virtue of the differing number 
of measures in each dimension, we re-ran the rank-distance analysis using differing numbers of 
measures in the financial, size, complexity and quality. Some analysis runs were similar, and 
some quite divergent in terms of proximity to the reference institution. Overall, there is 
enough variation in the resulting peer set to give one pause about relying solely on this method 
of extracting peer institutions. 

To address these concerns, we employ factor analysis (varimax rotation, and Kaiser's 
criterion for Eigenvalue selection) and a cluster analysis technique (complete linkage, 
hierarchical aggolomerative). The factor analysis uses principle components analysis to reduce 
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the original forty-three measures to eleven factors that reflect institutional dimensions of 
import. By definition, principle components analysis partitions the observed variance in the 
measures to unique factors that are completely uncorrelated. The researcher can though 
influence factor composition by choosing the extraction technique, and in deciding where to 
halt factor formation, so some subjectivity is still present, albeit in a limited manner. The 
dimensions that surfaced in this analysis are characterized as reflecting: Size and wealth; Socio- 
economic status; Student body complexity; Faculty and support; Transfer emphasis; Business 
and Liberal Arts & Science emphasis in associate degree programs ; Academic support. Student 
services support; non-Traditional emphasis; Business certificate emphasis; and Liberal arts and 
sciences emphasis in certificate programs. The factor scores are exported and saved in the 
dataset, and are then used in a cluster analysis. The progression of institutional clustering via a 
dendogram diagram can then be examined to determine how institutions cluster with the 
target institution. 

Each of the factors receives equal weight in building the clusters, and complete data is 
required for each institution. Initially, a conscious decision was made not to weight the factors, 
as no design induced implicit weighting scheme existed. After the first pass through, a 
weighting scheme was considered as senior campus officials believed that certain institutional 
or student body characteristics were in fact more important than others to the college mission 
and operations. Based on conversations with senior staff, it was decided to weight the 
dimensions into high, medium, and low importance categories. While a considerable amount 
of time and energy could have been invested in estimating more precise weights, given that 
these were largely 'opinion' data in the first place, the high, medium, and low classifications 
were deemed satisfactory. The dimensions and their weights were thusly defined: 


Size, 

Wealth 

Socio 

economic 

Student 

Body 

Complexity 

Faculty 

and 

Support 

Transfer 

Emphasis 

Bus/ 

Liberal 

Arts 

Acad 

Suppport 

Student 

Services 

Support 

Non- 

Traditional 

Emphasis 

Bus 

Certificate 

Emphasis 

Lib Arts 
Certif 
Emphasis 












High 

High 

Medium 

Medium 

High 

High 

High 

Low 

High 

Low 

Low 

1.00 

1.00 

0.66 

0.66 

1.00 

1.00 

1.00 

0.33 

1.00 

0.33 

0.33 


The cluster analysis calculates a standard Euclidean distance measure for each 
institution based on the standardized factor scores multiplied by their respective weights, and 
uses the "complete linkage" hierarchical agglomerative technique to group institutions into 
relatively homogeneous clusters. The clusters are formed based on the minimum maximum 
distance score between institutions, which is compared at each successive step until the 
researcher decides to stop cluster formation (based on professional judgment about 
diminishing returns). Each institution's cluster can be as small as two campuses, or can be built 
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larger to incorporate larger numbers, but at each successive step, the number of institutions 
that join clusters is variable, and depends on the clustering algorithm, the weights assigned to 
factors (or not) and the researcher's objective. 

The underlying purpose of this analysis is to identify a small number of institutions that 
group closest to the target institution. Institutions can be grouped by use of a dendogram, 
which traces the clustering pattern of any institution to successive institutions or groups of 
institutions. By definition, similarity between institutions becomes less distinct as the clusters 
incorporate additional schools. A dendogram is then produced based on the underlying 
weighted factors and the distance between institutions on them. Visual inspection is then used 
to determine where to stop the clustering process as more and more institutions join the initial 
cluster. 

Results 

Univariate statistics for the forty-three measures used are reported in Appendix A. Even 
though only public community colleges in New York State are included in the analysis, a brief 
review of the means and standard deviations suggests a great deal of variability on many of the 
measures across these institutions. This further reinforces the need for a comparison strategy. 
Unlike our prior explorations (1989, 1996, 2006) to identify peer institutions for public research 
universities that showed that the financial measures exhibited the most variability, followed by 
the size and then quality measures, in this analysis, there is considerably across these 
measures. 

The (truncated) bivariate results reported in the correlation matrix in Appendix B 
indicate that measures, in general, are highly correlated with each other, and reinforces the 
concerns noted above around redundancy of measures. For example, core revenues have a 
Pearson's correlation coefficient above 0.50 with several other financial measures. 

Furthermore, it is also highly correlated with freshman retention, the graduation rate, and the 
percentage of the student population that is minority. Other measures were also highly 
correlated both within and across the hypothetical dimensions of finance, size, quality, and 
complexity. While more in-depth discussion of these relationships is not entertained here due 
to space limitations, the important implication is that a factor analytic technique that controls 
for multicolinearity is a suitable approach to effectively understanding this data set. That said, 
the rank distance method is still seen as beneficial in terms of educating campus decision 
makers about relevant data and its spread among institutions. 

The second method used to develop a set of peer institutions is a factor and cluster 
analysis technique. When the forty-three measures described above are subjected to principal 
components analysis, with varimax rotation and using Kaiser's criterion for Eigenvalue 
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selection, eleven factors emerge which explain eight-five percent of total variance in the data 
set. Appendix C shows the factor loadings and the resulting dimensions. 

These results suggest that size and wealth measures largely load along a similar 
dimension (size and wealth) that explains the most variance (26.8%) of all the eleven 
dimensions extracted. The Socio-economic factor, comprised of measures such as the 
percentage of full-time undergraduates on Pell grants, median county income, the percentage 
of core revenue provided from tuition and fees, and the unemployment rate explain just over 
ten percent (10.9%) of the variance in the dataset. The remaining factors explain from 7.4 to 42 
of the dataset variance. 

As noted earlier, after an initial exploratory cluster analysis was conducted without 
factor weights, it was decided that using weights might provide a better acceptance of the 
clustering solution by campus-decision makers. 

Appendix D illustrates the dendogram that graphically depicts the institutional clustering 
sequence that results when the unweighted factor scores are submitted to the clustering 
algorithm. Appendix E depicts the resulting dendogram when weighted factors are used. Of 
primary interest in this exercise are those institutions that cluster with Columbia-Greene, the 
target institution. Examining the dendogram in Appendix E shows that Columbia-Greene first 
clusters with University of Adirondack CC and Jefferson CC. These two institutions are then 
joined by Cayuga CC. At the next iteration. Finger Lakes CC, Ulster CC, and Broome CC join this 
small cluster of four institutions. Then Jamestown CC joins this group, followed by Corning CC. 
Then the CUNY schools of Bronx CC and Hostos CC join the cluster. At the next iteration eight 
schools join the growing cluster, followed by a larger cluster composed of thirteen schools, then 
by two, and then by a final single institution, and all schools have been clustered. 

The decision the researcher must make at this point is to determine where to stop the 
clustering. In this exercise, the decision was made to group the clusters at the stage right 
before the CUNY schools would have joined in what is a cluster of SUNY schools, rounded out 
by Jamestown CC and Corning CC. This seems a natural stopping point, due to the inherent 
differences between governance and funding structures between the SUNY and CUNY systems. 
And further supported by the fact that if the CUNY schools were included, it would not be 
reasonable to include additional schools, as the cluster to form after they join would bring in an 
additional eight schools, making the peer set too large at eighteen institutions. 

Now that a cluster of peer institutions has been formed (Adirondack, Jefferson, Cayuga, 
Finger Lakes, Ulster, Broome, Jamestown, and Corning CCs), we might ask "how it might best be 
described?" These schools range in size from 1,400 to 5,100 students, with from 43 to 137 full- 
time tenured, tenure-track faculty. They serve similarly sized populations on non-traditionally 
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aged students, but have varying levels of non-degree students. In addition, their operating 
budgets run from twenty to over fifty million dollars. A more empirical approach would be to 
examine the original percentile rank measures for the peer cluster. One could also examine 
institutional rank on the factor scores themselves. 
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Table 3 - Rank on Individual Metrics for Cluster Peers 



Financials 


Size 


Complexity 


Quality 


Rnk- 

Expend_ 

Tot 

Rnk - 
Core_Rev 
enues 

Rnk- 

Core_Rev 

_Loc_app 

J3Ct 


Rnk - 
FTE_enr 
oil 

Rnk - 
FT_TTrac 
k Fac 


Rnk - 
UGj3Ct_ 
PT 

Rnk - 
pct_age_ 
2565 

Rnk - 
pct_FTFT 
_ug_PEL 
L 

Rnk - 
Stu_pct 
min 

Rnk - 
Deg_Asso 
cj3ct_tec 
hnical 

Rnk - 
median 
income 


Rnk - 
FTretention 
rate 

Rnk - 
Rnk* Trans_o 
Grad rate ut 

SUNY Columbia'Greene 

33 

33 

4.5 


34 

33 


10 

7 

23.5 

25 

20 

15 


16.5 

9 24.5 


















SUNY Adirondack 

30 

30 

20 


27 

21 


18 

8.5 

26 

35 

22 

13 


23.5 

22 22 

SUNY Jefferson CC 

29 

26 

24 


29 

25 


19 

5.5 

17.5 

24 

26 

30 


28.5 

9 18 


















SUNY Cayuga CCC 

20 

21 

24 


26 

31 


11 

5.5 

5 

33 

17 

20 


19.5 

20 32 


















SUNY Finger Lakes CC 

22 

17 

32 


20 

17 


12 

24 

20 

28 

28 

17 


28.5 

3.5 18 

SUNY Ulster CC 

28 

28 

10.5 


30 

26 


2 

27 

27 

21 

34 

9 


19.5 

6 34.5 

SUNY Broome CCC 

19 

19 

10.5 


17 

13 


35 

17 

25 

31 

2 

27 


16.5 

7 34.5 

















SUNY Jamestown 

26 

24 

17.5 


25 

24 


34 

1.5 

10.5 

27 

32 

32 


13.5 

1 32 

















SUNY Corning CC 

24 

25 

7.5 


24 

22 


5 

24 

15 

32 

9 

26 


23.5 

16.5 24.5 


Alternative ways of looking at the inherent attributes of the institutions in the peer 
clusters is to graph the factor scores themselves. Chart 1 below uses the mean factor scores to 
examine three of the institutional clusters from the eleven cluster final I solution used to 
determine the nine member Columbia-Greene peer group described above. The first cluster 
represents Columbia-Greene and the schools listed in Table 3 above. The second cluster 
includes a grouping of three schools: SUNY Orange CC, SUNY Dutchess CC, and SUNY Rockland 
CC. The third cluster includes a grouping of six schools: SUNY Erie CC, SUNY Hudson Valley CC, 
SUNY Mohawk Valley CC, SUNY Monroe CC, SUNY Onondaga CC, and SUNY Niagara CC. Quick 
visual inspection shows that the Columbia-Greene cluster (1) is indeed different from the other 
two clusters on the first three factor dimensions. Differences exist among all of the eleven 
clusters, but only three were chosen for this demonstration to reduce visual complexity. Of 
import is the fact that these particular dimensions were chosen for inspection because between 
them they explain nearly 50 percent (45.1%) of the variance in the institution dataset. 

The Columbia-Greene cluster has a considerably lower value on the size and wealth 
dimension than the second cluster, and considerably less than the third cluster. While closer to 
average on the socio-economic dimension, the Columbia-Greene cluster also exhibits lower 
values than the other two clusters on this dimension, but the second cluster of schools has 
considerably higher values on this dimension than either of the other two clusters. Finally, 
while the student body complexity of Columbia-Greene's cluster is about the average value for 
all schools in the dataset, the schools in the second cluster have lower values, on average, and 
the schools in the third cluster have higher values on the student body complexity measure, on 
average. 
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Chart 1 - Mean factor scores for three clusters of institutions 


2 

1.5 

1 


0.5 


0 

- 0.5 



-1 
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■ Average of wt_F_Size_Wlth ■ Average of wt_F_SociEco ■ Average of wt_F_StuBody 


An alternative to looking at cluster differences to build confidence in the peer group 
candidates is to look at the attributes of the institutions that are in within the cluster that 
includes the target institution and highlight their similarities. Chart 2 below shows the mean 
factor scores for the nine institutions in the final Columbia-Greene peer cluster. Once again, 
only the first three dimensions of size and wealth socio-economic status, and student body 
complexity are examined in this particular chart. 

Chart 2 - Within Cluster Attributes 



■ Sum of wt_F_Size_Wlth ■ Sum of wt_F_SociEco ■ Sum of wt_F_StuBody 
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It is important to keep in mind that the factors scores were weighted in the final 
solution. Visual inspection seems to confirm that these institutions are more homogeneous 
than heterogeneous on these dimensions. While less similarity is observed on the student body 
complexity dimension than on the size and wealth or socio-economic dimensions, we must 
keep in mind that the student body complexity dimension only had 'medium' weight, while the 
other two had 'high' weights. And the setting of weights is a subjective decision made by 
campus leaders. 

Discussion 

In summary, the rank distance and factor and cluster analysis techniques provided an 
overlapping group of peer institutions for the target institution, Columbia-Greene Community 
College (C-GCC). In both analyses, campus decision makers have opportunity to play an integral 
role in developing the list of variables and their respective measures used to differentiate 
institutions. With both techniques, the institutional research staff, as well as higher level 
campus decision makers, can clearly see how C-GCC compares to other institutions across 
financial, size, complexity, and quality measures. Increased familiarity with and knowledge 
about these aspects of one's own institution and its chosen peers is of importance as we move 
forward with programmatic initiatives. Benchmarking to external institutions is often 
requested/expected by governing bodies, and having an up-to-date set of benchmark 
institutions adds confidence to the exercise. 

There are limitations to these methods. The rank distance method employed did not 
assign weights to the individual measures. While an implicit weighting scheme was in effect due 
to the different number of measures used to represent finance, size, quality, and complexity 
variables, arguments can certainly be made to add measure weighting, or to alter the number 
of measures. As demonstrated, the number of measures used can indeed have an impact on 
the result set. In the past, analyses were guided by a desire to seek peer institutions more 
closely related on the financial and size variables, and this analysis was conducted similarly. We 
will need to revisit this approach as we work with campus leaders to develop a formal set of 
peer institutions. 

Another limitation of the rank distance method is that basing the distance measure on 
percentile ranks instead of on factor or standardized scores potentially minimizes the 
magnitude of the differences between institutions. The percentile rank approach creates a 
distance measure based on the rank order of institutions for different measures rather than 
upon the magnitude of their differences from each other. The actual distances in the raw data 
may be greater or smaller than the distance between ranks. 
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A final limitation of this methodology worth noting is that many of the measures used, 
as noted, are highly correlated with each other. In other words, to a certain degree, they may 
be measuring the same institutional aspects. While the measures used address different 
concerns campus decision makers harbor, the end result may be confounded by the highly 
related nature of the measures. 

The factor and clustering technique, which alleviates the problem of highly correlated 
measures by factoring them into eleven completely uncorrelated dimensions has an advantage 
in this respect. In this analysis, we decided to use weights on each of the eleven dimensions to 
provide for a more robust clustering solution. 

A well noted limitation of factor analysis is that the factors (dimensions) used for 
clustering are more difficult to comprehend. This is particularly true with respect to the 
percentile ranks used in the first analysis. While standardized factor scores do not easily lend 
themselves to meaningful interpretation, once can still compare factor scores across 
institutions, or by cluster, to convey a sense of similarity/dissimilarity. 

Both of these methods provide a means of identifying peer institutions. Neither should 
be viewed as a turn-key approach. The rationale for the undertaking in the first place plays a 
significant role in choosing variables and measures of them for consideration. Acceptance of a 
peer institution group by campus decision makers and relevant external audiences largely 
depends on accommodating and incorporating their intuition, concerns, and political 
objectives. The peer review process is a learning process. Through it we learn not only about 
the funding, quality, or size of other institutions, but more importantly, we learn about how our 
own institution stands in those respects. 

Future Research 

Future research might address the methodological limitations noted above. These 
include important methodological aspects, such as the sources of variables and selection of 
variables — that in turn strongly influence the composition of the eventual peer groupings. 

Due to resource, accessibility, and other constraints, this study has focused only on data 
available through the IPEDS database. However, there are other reliable sources of institutional 
data from sources such as the National Research Council (NRC), and the National Science 
Foundation (NSF) which can provide additional measures for peer analyses. Looking to the 
future, these sources can be used to incorporate variables that better measure, for example, 
the quality dimension through faculty work products such as the number of publications per 
faculty or the number of citations per faculty. Other possibilities may exist as well. 
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Appendix A - Univariate Statistics 



Mean 

Std. Deviation 

Anaiysis N 

Missing 

N 

Complexity 





Core_Expenses 

45.91 

8.31 

35 

0 

Core_Exp_lnstruct_pct 

826 

3.12 

35 

0 

Core_Exp_Acad_Suptjct 

10.37 

2.53 

35 

0 

Core_Exp_Stu_Serv 

18.74 

4.60 

35 

0 

Core_Exp_lnstl_Supt_pct 

6.915.14 

5,130.87 

35 

0 

Core_Revenues 

84,838,32100 

63,274,421.83 

35 

0 

Core_Rev_Tuit_Fees_pct 

20.17 

6.02 

35 

0 

Core_Rev_Slate_app_pct 

2106 

4.54 

35 

0 

Core_Rev_Loc_app_pct 

19.11 

6.06 

35 

0 

Tot_Emp_FT 

456.34 

298.42 

35 

0 

Tot_Emp_PT 

547.97 

463.57 

35 

0 






Size 





FTE_enroll 

84,992,931.57 

64,536,164.12 

35 

0 

FT_TTrack_Fac 

135.86 

103.90 

35 

0 

FT_Fac_lnstrLect 

34.06 

29.69 

35 

0 






Complexity 





FTFT_pct_of_Ugs 

19.23 

3.83 

35 

0 

Transfers_pct_of_new 

21.05 

5.31 

35 

0 

UG_pct_PT 

40.71 

6.91 

35 

0 

pct_enrl_nondeg 

2028 

10.92 

35 

0 

NJCAA 

0.97 

0.17 

35 

0 

pct_age_18_24 

5838 

6.51 

35 

0 

pcl_age_2565 

27.50 

3.41 

35 

0 

pct_FTFT_ug_PELL 

60.69 

13.12 

35 

0 

pct_FTFT_st_loc_grant_aid 

57.26 

12.23 

35 

0 

pct_FTFT_insa_grant_aid 

5.40 

7.36 

35 

0 

res_halls 

0.29 

0.46 

35 

0 

Stu_pct_min 

29.24 

25.36 

35 

0 

Fac_pct_FT 

36.03 

14.04 

35 

0 

Deg_Assoc_tot 

1,13829 

838.06 

35 

0 

Deg_Assoc_pct_technical 

4.00 

2.93 

35 

0 

Deg_Assocjct_lib_arts 

4141 

9.25 

35 

0 

Deg_Assoc_pct_Soc_Setv 

30.92 

8.43 

35 

0 

Deg_Assoc_pct_Bus 

16.46 

7.88 

35 

0 

Certif_1_2yrs_Tot 

68.00 

72.17 

35 

0 

Cert_pct_technical 

1149 

12.26 

35 

0 

Cert_pct_lib_arts 

14.56 

20.10 

35 

0 

Cert_pct_Soc_Setv 

51.59 

31.02 

35 

0 

Cert_pd_Bus 

10.46 

19.31 

35 

0 

unemploy_rate 

939 

1.74 

35 

0 

county_pop 

627,030.83 

714,121.81 

35 

0 

medianjncome 

53,803.69 

13,970.71 

35 

0 






Quality 





FT_retention_rate 

60.03 

7.04 

35 

0 

Grad_rate 

21.14 

6.37 

35 

0 

Trans_out 

19.03 

3.91 

35 

0 
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Appendix B — (truncated) Correlation Matrix (a complete correlation matrix is available from the authors, upon request) 



Core_Exp 
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Core_Exp 
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pet 

Core_Rev 
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Core_Rev 

_Loc_app 

_pct 

FT£_enrol 

1 

FT_TTrack 

_Fac 

FT_Fac_ln 

strLect 

UGj3Ct_P 

T 

pct_enrl_n 

ondeg 

pct_age_2 

5-65 

pct_FTFT_ 

ug_PELL 

Stu_pct_m 

in 

Fac_pct_F 

T 

Deg_Asso 

cjot 

Deg_Asso 

c_pct_tech 

nical 

unemploy 

_rate 

countyjDO 
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medianjn 
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FT_retenti 

on_rate 

Grad_rate 

Trans_out 

Core_Expenses 

1 

.393' 

.991" 

0.18 

.975" 

.951" 

.742" 

-0.26 

-.594" 

0.04 

-0.09 

.641" 

0.19 

.941" 

0.24 

-0.27 

.831" 

.513" 

.607' 

-.517" 

0.01 

Core_Exp_lnstruc 

Lpct 


1,00 

.366' 

0.19 

.449" 

.431" 

0.17 

-0.06 

-0.25 

-0.04 

-.628" 

0.08 

0.22 

.399' 

0.32 

-.453" 

0.28 

.568" 

.486" 

-0.08 

0.05 

Core_Revenues 



1.00 

0.13 

.962" 

.941" 

.772" 

-0.25 

-.607" 

0.08 

-0.01 

.676" 

0.21 

.942" 

0.25 

-0.23 

,842" 

.447" 

.590" 

-.510" 

-0.01 

Core_Rev_Loc_a 

pp_pct 




1.00 

0.06 

0.20 

0.08 

-0.06 

-.343 

-0.17 

-0.14 

.407' 

0.26 

-0.01 

0.00 

0.11 

0.25 

0.29 

0.29 

-.358' 

-0.06 

FTE_enroll 





1.00 

.947" 

.730" 

-0.27 

-.530" 

0.05 

-0.18 

.511" 

0.12 

.978" 

0.26 

-.361* 

.768" 

.545" 

.597" 

-.409' 

0.07 

FT_TTrack_Fac 






1.00 

.784" 

-.341' 

-.610" 

0.01 

-0.08 

.608" 

0.16 

.912" 

0.33 

-0.24 

.808" 

.458" 

.588" 

-.470" 

0.03 

FT_FacJnstrLect 







1.00 

-0.27 

-.515" 

0.10 

0.11 

.682" 

0.22 

.738" 

0.30 

0.01 

.759" 

0.14 

.543" 

-0.31 

-0.03 

UG_pct_PT 








1.00 

.708" 

-0.16 

-0.06 

-0.11 

-0.10 

-0.32 

-0.32 

-0.05 

-0.13 

0.04 

-0.16 

-0.15 

0.03 

pct_enrl_nondeg 









1.00 

-0.32 

-0.06 

-.613" 

-.356' 

-.541" 

-0.26 

-0.08 

-.530" 

-0.18 

-.530" 

.336' 

0.17 

pct_age_2565 










1.00 

0.06 

0.04 

0.31 

0.06 

0.14 

0.05 

0.05 

-0.10 

0.10 

-0.08 

-0.30 

pct_FTFT_ug_PE 

LL 











1.00 

.334' 

0.00 

-0.14 

-0.06 

.646" 

0.15 

-.756" 

-0.32 

-0.10 

-0.13 

Stu_pd_min 












1.00 

.373' 

.479" 

0.06 

0.26 

.812" 

0.08 

.476" 

-.652" 

-0.07 

Fac_pct_FT 













1.00 

0.05 

0.21 

0.08 

0.24 

0.10 

0.04 

-0.33 

0.21 

Deg_AssocJot 














1.00 

0.24 

-.340' 

.743" 

.461" 

.580" 

-0.30 

0.03 

Deg_Assoc_pct_t 

echnical 















1.00 

-0.11 

0.18 

-0.02 

0.00 

-0.25 

0.12 

unemploy_rate 
















1.00 

0.05 

-.699" 

-0.32 

-0.02 

-0.32 

county_pop 

















1.00 

0.20 

.593" 

-.510" 

-0.10 

medianjncome 


















1.00 

.457" 

-0.29 

0.25 

FT_retention_rate 



















1.00 

-0.16 

-.365' 

Grad_rate 




















1.00 

-0.10 

Trans_ou1 





















1.00 
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Appendix C - Factor Matrix 


Rotated Component Matrix* 



Size, Wealth 

Socio 

economic 

Student Body 
Complexity 

Faculty and 
Support 

Transfer 

Emphasis 

Bus/ Liberal 
Arts 

Acad 

Suppport 

Student 

Services 

Support 

Non- 

Traditional 

Emphasis 

Bus 

Certificate 

Emphasis 

Lib Arts Certif 
Emphasis 

Pet of Variance Explained 

26.3 

10.9 

7.4 

5.8 

5.7 

5.2 

4.8 

4.8 

4.7 

4.3 

4.2 

Tot_Emp_FT 

MO 

.096 

021 

082 

.019 

.010 

-.031 

-.101 

.066 

.068 

088 

Core_Revenues 

.961 

.076 

.013 

.088 

-.002 

-.097 

-.064 

-.026 

.070 

-.001 

.073 

Expend_Tot 

,96t) 

.146 

.017 

.092 

.013 

-.066 

-.066 

-.011 

.037 

.011 

,104 

Core_Expenses 

.953 

.150 

.022 

.087 

.015 

-.060 

-.069 

-.011 

.042 

.011 

.112 

FTE_enroll 

.954 

.219 

.034 

-.015 

.088 

-.016 

.027 

-.041 

.084 

-.001 

.072 

Deg_Assoc_tot 

■ 1 ■ 

.157 

.082 

-.080 

.059 

-.017 

.081 

-.020 

.083 

-.014 

.021 

FT_TTrack_Fac 

.944 

.126 

.063 

.078 

.023 

.001 

-.066 

-.114 

-.007 

.029 

.093 

Tot_Emp_PT 

.934 

.128 

.028 

-.188 

-.074 

.018 

-.023 

-.021 

-.122 

-.033 

-.022 

county_pop 

.867 

-.122 

-.157 

.223 

-.082 

-.157 

-.100 

.067 

-.081 

.024 

.092 

FT_Fac_lnstrLect 

.834 

-.136 

-.068 

.075 

-.041 

-.027 

-.087 

-.042 

.174 

.134 

-.192 

Stu_pct_min 

.663 

-.258 

-.107 

.428 

-.162 

-.174 

-.328 

.152 

-.127 

.093 

-.016 

FT_retention_rate 

.637 

.354 

-.081 

.135 

-.452 

.001 

.212 

.010 

-.118 

-.072 

-.032 

pd_enr1_nondeg 

-.592 

.034 

-.495 

-.430 

.360 

-.100 

.079 

.032 

-.073 

-.021 

.066 

Deg_Assoc_pct_Soc_Serv 

-.565 

-.144 

-.013 

-.306 

.169 

.045 

-.353 

.082 

.130 

.402 

.214 

pct_FTFT_ug_PELL 

.025 

-MB 

.000 

.033 

-.017 

-.137 

-.052 

.053 

-.034 

-.106 

.085 

medianjneome 

.371 

.833 

-.028 

.092 

.097 

.072 

-.082 

.001 

-.133 

-.061 

.026 

pct_FTFT_st_loc_grant_aid 

-.164 

-.518 

.254 

-.187 

.206 

.103 

.023 

-.035 

-.065 

.018 

.094 

Core_Rev_Tuit_Fees_pct 

-.023 

.735 

.169 

-.347 

.134 

.080 

-.053 

.189 

.218 

.053 

.190 

unemploy_rate 

-.168 

-.729 

-.045 

.133 

-.258 

.107 

-.186 

.199 

-.021 

.206 

-.066 

Core_ExpJnstruct_pct 

.325 

.601 

-.043 

.271 

.047 

.024 

.292 

-.338 

-.082 

.189 

-.003 

FTFT_pct_Of_Ugs 

-.191 

-.010 

.935 

.025 

.115 

.016 

.074 

-.009 

-.034 

.054 

-.132 

UG_pCt_PT 

-.277 

.084 

-.853 

-.092 

.136 

-.144 

-.016 

.096 

-.084 

-.032 

-.055 

T ran sfers_pct_of_n ew 

.000 

-.002 

1 -.566 

.192 

-.014 

-.003 

-.099 

.171 

.149 

.074 

.474 

pd_age_18_24 

.445 

.384 

; .564 

.138 

-.085 

.256 

-.253 

.100 

.093 

.241 

.112 

res_halls 

-.300 

-.270 

.548 

-.224 

.372 

.054 

-.044 

.145 

.126 

.115 

.168 

Fac_pd_FT 

.083 

-.010 

.001 


.151 

.021 

-.040 

-.026 

.258 

.075 

-.062 

pd_K K_instl_grant_aid 

-.393 

-.074 

-.173 

-.546 

.020 

.017 

-.100 

.366 

.257 

.136 

.247 

Core_Rev_Loc_app_pd 

.110 

.156 

-.062 

.466 

-.256 

.363 

-.315 

.029 

-.459 

.232 

.251 

Trans_out 

-.023 

.143 

.140 

.169 

.848 

-.013 

-.150 

.075 

-.094 

,011 

-.163 

Cofe_Exp_lnstl_Supt_pd 

-.057 

-.159 

.200 

.046 

B -.597 

.142 

-.320 

-.015 

-.083 

,167 

-.225 

Deg_Assoc_pd_Bus 

273 

-.038 

-220 

.061 

,162 

-.811 

.074 

.041 

-.001 

-.195 

-.053 

Deg_Assocj5d_lib_arts 

.171 

.045 

.069 

V68 

-.293 i .742 

.262 

.187 

-.096 

-.232 

-.116 

NJCAA 

-.235 

.066 

.009 

-.047 

iYot .666 

-.010 

-.002 

.064 

-.127 

-.155 

Core_Exp_Acad_Supt_pd 

-.201 

.035 

.038 

.033 

-.099 

.032 

752 

.311 

.254 

-.054 

-.011 

Grad_rate 

-.425 

.004 

.284 

-.349 

-.051 

.075 

.587 

.137 

-.043 

.012 

-.129 

Core_Rev_State_app_pd 

.014 

.400 

-.176 

.014 

.355 

.122 

.509' 

-.223 

.009 

.196 

.031 

Core_Exp_Stu_Serv 

.103 

-.164 

.029 

-.003 

.246 

.123 

.107 

M2 

.079 

.127 

-.023 

Deg_Assoc_pd_technical 

.217 

-.015 

.157 

094 

.166 

.039 

-.176 

I -.767 

.280 

.099 

.031 

Certif_1_2yrs_Tot 

.252 

.296 

.013 

-020 

.193 

-.081 

.115 

-.106 

731 

.243 

.092 

pd_age_2565 

.013 

-.049 

.018 

.276 

-.364 

.064 

.048 

-.002 


-.293 

.044 

Cert_pd_technicai 

.092 

-.128 

.024 

.061 

.051 

.388 

.'208 

-.425 

i .467 

.276 

-.072 

Cert_pd_Bus 

-.103 

.000 

-.116 

-.081 

.023 

.023 

.005 

.001 

.003 

.8M 

-.003 

Cert_pd_lib_arts 

-.341 

.126 

.053 

.137 

-.091 

.070 

.062 

.109 

.006 

.134 

-798 

Cert_pd_Soc_Serv 

-.003 

.249 

-.033 

-.078 

-.199 

-.201 

.062 

.038 

.014 

.491 
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Dendrogram using Complete Linkage 

Rescaled Distance Cluster Combine 


Appendix D - Dendogram, unweighted Factors 
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SUNY Westchester CC 




Dendrogram using Complete Linkage 

Rescaled Distance Cluster Combine 


Appendix E - Dendogram, Weighted Factors 
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The National Survey of Student Engagement (NSSE), an online survey administered to 
freshmen and seniors, can be used to measure the quality of student experiences. NSSE survey 
items, described by NSSE as representing “empirically confirmed good practices,” relate to 
individual student behaviors as well as student perceptions of their college experience. 
Institutions — over 750 last year — use NSSE to support decision making, design goals, and 
analyze progress in such areas as accreditation, accountability reporting, strategic planning, and 
program assessment (Banga, Pike, & Hansen, 2009). 

NSSE can be a valuable tool, but with a national response rate of approximately 27% and 
roughly one third of institutions achieving response rates below 30% (NSSE, 2011), there are 
concerns regarding the degree to which institutions can generalize their results to their student 
population. Eow response rates do not necessarily suggest a lack of representativeness, however; 
a survey yielding a low response rate may still produce a sample that is representative of its 
population (Dey, 1997; Groves, 2006). It is only when there are discrepancies between 
responders and nonresponders in perceptions or behaviors relevant to the survey topic that 
nonresponse bias is introduced and becomes a threat to the validity of inferences made from the 
survey results. NSSE acknowledges the impact that nonresponse bias may have on the 
interpretation of an institution’s results and therefore encourages institutions to conduct their 
own nonresponse studies (Chen, 2006). 

In short, our purpose in conducting the present study was to meet NSSE’s call for 
institutional-level nonresponse studies. Our institution, the University of Maine, is a land-grant 
university with a total student population of approximately 1 1,000 (roughly 8,270 of whom are 
degree-seeking undergraduates). UMaine administered the web-based NSSE to all freshmen and 
seniors in spring 2011, achieving a response rate of approximately 21%. To determine the 
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degree to which these respondents are representative of the remaining 79% of freshmen and 
seniors, we addressed the following questions: 

a) How do the demographic characteristics of nonresponders compare with those of the 
responders? 

b) What reasons for nonresponse are subsequently offered by nonresponders? 

c) How do nonresponders’ perceptions of their UMaine experience compare with those of 
responders, using a selection of items from the NSSE survey? 

Related Literature 

NSSE Overview 

NSSE emerged from the efforts of a design team, lead by Peter Ewell of the National 
Center for Higher Education Management Systems, that had been tasked by Pew Charitable 
Trusts to develop an instrument to measure the extent to which college students show good 
educational practices and to assess what they gain from their college experience (Kuh, 2009). 
The survey was first administered to 276 institutions in 2000; in 2011, 751 institutions from the 
United States and Canada participated (NSSE, 201 1). Institutions use NSSE to identify areas 
where institutional policies or practices may be improved to promote good educational practices, 
and to serve as an external accountability measure of overall quality (NSSE, 2011). The 
Voluntary System of Accountability (VSA), an initiative through which institutions report 
measures of student outcomes, chose NSSE as one of the options schools can use for assessing 
student engagement. NSSE data, along with other information provided in the spirit of 
accountability and public disclosure, appear conspicuously on the participating institution’s 
website, in the “College Portrait.” In addition to engagement data, the College Portrait provides 
general consumer information as well as measures of student success and progress, learning 
outcomes, and perceptions and experiences. 



4 


The three core purposes of the NSSE are to provide actionable data to institutions so that 
they can improve the undergraduate experience, highlight and document effective educational 
higher education practices, and advocate for public acceptance and use of standard measures for 
college quality (Kuh, 2009). Through five sets of questions, students are asked about (a) their 
participation in sports and activities, study time, interaction with faculty members; (b) what they 
see their institution as requiring of them; (c) their perceptions of their college environment 
(including factors believed to impact satisfaction, academic achievement, and persistence); (d) 
information regarding their socioeconomic and educational background; and e) their educational 
and personal growth since starting at the institution. 

For ease in analysis and interpretation, NSSE developed five benchmark scales of 
effective educational practice that capture institutional characteristics and student behavior 
highly aligned with both learning and personal development. Comprising a total of 42 items, the 
five benchmark scales are Eevel of Academic Challenge, Active and Collaborative Eeaming, 
Student-Faculty Interaction, Enriching Educational Experiences, and Supportive Campus 
Environment (Kuh, 2009). 

Although NSSE is a well-known and frequently used survey, validity concerns have been 
raised regarding the accuracy of student self-reports (Porter, Rumann, & Pontius, 201 1), the 
weak relationship with student academic outcomes (Gordon, Eudlum, & Hoey, 2008), and low 
response rates (Porter & Whitcomb, 2005). In light of these concerns, institutions should take 
steps to verify the validity of the data collected at their own institution. In fact, Chen et. al. 
(2009), in their outline of best practices for analyzing NSSE data, suggest that determining the 
quality of an individual institution’s data should be a priority for any analyst using NSSE data. 
They stress the importance of verifying that population estimates are accurate, and recommend 
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that institutions consider sampling error, analyze the potential for nonresponse bias, and examine 
the proportional representation of student subgroups within their responder samples. 

Nonresponse bias in survey research 

Over time, survey response rates have suffered a noticeable decline (Dey, 1997). As a 
result, the awareness of and concerns regarding nonresponse bias has also grown in recent years. 
The following is a more comprehensive definition of nonresponse bias and a summary of the 
methods used to examine its existence. 

Definition of nonresponse bias. As mentioned above, the literature on nonresponse bias 
suggests that bias is not necessarily a function of low response rates. Rather, it is only when 
there is a discrepancy between respondents and nonrespondents on relevant attitudes, 
perceptions, and behaviors that bias is introduced. Low response rates then serve to exacerbate 
the level of bias (Groves, 2006; Pike 2008). 

Nonresponse bias, and the degree to which it is a problem, is often related to the reasons 
for nonresponse (Groves, 2006; Rogelberg & Luong, 1998). Rogelberg and Luong (1998) 
outline the four major classes of nonresponse as inaccessibility, inability to respond, 
carelessness, and noncompliance. Each reason is likely to affect bias in a different way, 
something that Groves (2006) stresses should be taken into account by survey researchers. For 
example, nonresponse due to an inability to contact a person is likely to introduce a different 
level of bias than nonresponse because of a refusal to answer questions about the particular 
subject. Both reasons will have an impact on response rate, but biased responses are more likely 
with the latter. The former may be completely unrelated to the attitudes, behaviors, or 
perceptions of interest in the survey. 

Thus, there are four important considerations when interpreting survey results: response 
rates across various subgroups in the population of interest; demographic differences between 
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responders and nonresponders; distinctions between responders and nonresponders in the 
attitudes, perceptions, or behaviors measured by the survey; and reasons for nonresponse. The 
first two can be examined using existing data that are often readily available, but one must 
collect data from the nonresponder group in order to address the latter two concerns. 

Methods for examining nonresponse bias. Nonresponse bias can be examined using a 
number of methods. There are three approaches researchers regard most promising for studying 
nonresponse (Dey, 1997; Groves, 2006; Hartman, Fuqua, & Jenkins, 1986; Porter & Whitcombe, 
2005; Rogelberg & Luong, 1998). In the archival approach, one compares nonresponders and 
responders on a number of relevant demographic variables and then examining each variable’s 
relationship with the survey responses. In a wave analysis, one compares early and late 
responders, assuming that late responders are most similar to nonresponders. And in Si follow-up 
analysis, one obtains responses from a sample of nonresponders and then examines the degree to 
which the two groups differ on survey items of interest. 

These three approaches have both strengths and weaknesses. For example, although the 
analysis of demographic variables used in the archival approach can be conducted with a simple 
matching of data, the information provided only gives a partial indicator of the extent to which 
bias exists; because it does not provide any information on nonresponders (Groves, 2006). A 
wave analysis is also relatively easy to conduct, particularly if the survey was administered 
through a process of multiple reminders, but it assumes that nonresponse bias will be 
systematically distributed over time (Hartman, Fuqua, & Jenkins, 1986). Because of these 
considerations, the follow-up analysis is a common approach to understanding nonresponse. Its 
primary benefits are that it allows for analyses of the attitudes and behaviors of both responders 
and nonresponders (Porter & Whitcombe, 2005) and therefore can provide a basis for estimating 
their differences — and, therefore, the extent of any bias (Hartman, Fuqua, & Jenkins, 1986). 
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That said, the follow-up approach is not without its weaknesses. Its primary weakness 
(ironically) is the risk of additional bias introduced with the choice of method for data collection. 
If the method used to collect responses from the follow-up sample distorts who responds and the 
answers they furnish, the responses will not provide an unbiased view of nonresponders (Dey, 
1997; Porter & Whitcombe, 2005; Rogelberg & Luong, 1998). For example, telephone surveys 
may produce biased results (Dillman, Sangster, Tamai, & Rockwood, 1996). Having reviewed 
the literature on the comparisons of responses to mailed or phone surveys, Dillman et. al. 
propose that telephone interviews are more likely than mail questionnaires to produce (a) 
socially desirable and acquiescent answers, (b) question-order effects, (c) quick answers that 
reflect a general standard held by the respondent, and (d) extremeness on response scales. These 
limitations call for caution when interpreting follow-up studies conducted through telephone 
surveys. 

Analyses of nonresponse bias in NSSE Surveys 

NSSE researchers have used both the archival and follow-up approaches to examine 
nonresponse. 

Archival approach. NSSE researchers have consistently found significantly lower 
response rates among men and part-time students than women and full-time students (NSSE, 
2011). NSSE uses a weighting procedure in their institutional report to compensate for such 
differences. 

In their multi-level study. Porter and Umbach (2006) used the NSSE to examine how 
response rates vary by the makeup of the student body and institutional characteristics. With a 
sample of approximately 167,000 students across 321 schools, these researchers also found 
women were more likely to respond than men. In addition to differences associated with gender. 



their results showed that student ability and such social environment factors as density, urbanity, 
and the percentage of part-time students also affected institutional response rates. 

NSSE (2008) examined the relationship between levels of high school engagement, as 
measured by the Beginning College of Student Engagement, and whether or not a student 
responded to the NSSE in the spring of the freshman year. Based on data from approximately 
35,000 students across 89 institutions, the analyses revealed no relationship between high school 
engagement and propensity to respond. 

Follow-up studies. Two national NSSE follow-up studies suggest the existence of 
nonresponse bias. In a 2001 study, the Indiana Center for Survey Research (CSR) conducted a 
nonresponse analysis based on follow-up telephone surveys with 553 nonresponders from 21 
institutions (Kuh, 2003). The interviews included 21 engagement and 3 demographic items from 
the survey. Ereshmen nonresponders scored higher than respondents on nine items, while 
responders only scored higher on three items. Senior nonresponders scored higher than 
respondents on six items, and responders scored higher on the same three items seen in the first- 
year group. In general the results showed a slightly higher level of engagement among 
nonresponders than responders (although the CSR researchers acknowledged the need for 
caution in interpreting their results due to the potential bias introduced by the use of a telephone 
interview). 

We see somewhat similar results in a later study, also conducted by the CSR (NSSE, 
2006). The second study included phone interviews with 1,408 nonresponders from 24 different 
institutions. The telephone interviews included 17 questions, with items representing student- 
faculty interaction, the campus environment, and developmental-gain subscales. Eour 
demographic items also were included. The results of the telephone interviews suggest 
nonresponders were more likely than respondents to view faculty, staff, and their campus as 
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supportive. However, these two groups did not differ in student-faculty interaction or 
developmental gains. 

Mclnnis (2006) used an additional mailing of the survey itself rather than a telephone 
survey to reach first- year nonresponders. Mclnnis received surveys from 25 of 94 nonresponders 
(26.6% response rate). Similar to the previous two NSSE studies conducted by CSR there were 
minimal differences. The only scale that showed a significant difference was the faculty 
interaction scale, with nonrespondents showing higher mean scores than respondents. 

The results of the national nonresponse bias studies conducted by CSR and the small- 
scale study conducted by Mclnnis (2006) suggest there is potential for nonresponse bias to 
threaten the generalizability of NSSE results. Although each study suggests a slightly higher 
level of engagement in some areas among nonresponders, the threat of bias associated with 
NSSE’s use of follow-up telephone surveys and the small scale of the Mclnnis study do not 
provide strong evidence that the same differences may be present at all schools. Again, the 
degree to which such a bias exists is likely to differ across institutions. As we consider the 
results of the 2011 NSSE results for UMaine, it therefore is helpful to know how responders may 
differ from nonresponders, in terms of both their demographic characteristics and their 
perceptions of the UMaine experience. 

Method 

As reported early the three research questions we addressed in this study of nonresponse 
bias are: 

(a) How do the demographic characteristics of nonresponders compare with those of 
the responders? 

(b) What reasons for nonresponse are subsequently offered by nonresponders? 
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(c) How do nonresponders’ perceptions of their UMaine experience compare with 
those of responders, using a selection of items from the NSSE survey? 

We used the archival approach to answer the first question and a follow-up analysis to address 
the second and third questions. 

Archival approach. We began by matching NSSE’s list of responders and 
nonresponders to UMaine’s spring 201 1 student database. We then created a dataset comprising 
these student characteristics: gender, residency (Maine resident, nonresident), enrollment status 
(part-time, full-time), cumulative GPA (below 2.5, 2. 5-2.9, 3. 0-3. 4, 3.5 and above), transfer 
status (new student, transfer student), living situation (on-campus, off-campus), and college of 
major. We used the x test of independence to determine whether there was a relationship 
between NSSE participation and the respective demographic variables. 

Follow-up analysis. To assess the perceptions of NSSE nonresponders, we conducted 
brief telephone interviews with 50 freshmen and 50 seniors whom we randomly sampled from 
the list of UMaine’s nonresponders. Its possible biases notwithstanding, we chose this method 
because of the short timeframe it permitted and its relatively inexpensive cost. 

To contact the students, we pulled random samples of approximately 50 freshman and 50 
senior names and telephone numbers from the nonresponder list provided by NSSE. We called 
students during both day and evening hours to reach working and non- working students alike. In 
the event that a student was not home, we did not leave a message (to relieve the student of the 
burden of calling back). We attempted to contact each student in our initial sample three times. 
Once we had either reached or exhausted three attempts to contact each student, we repeated the 
process with three additional random samples of 50 freshmen and 50 seniors. In total, we 
attempted to reach approximately 370 students. The data collection occurred between June 1, 


2011 and August 11,2011. 
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To keep the interviews to two to three minutes, we asked only five questions of the 
students. We began by asking why they did not respond to the NSSE, followed by four 
questions taken verbatim from the survey: 

(a) How would you rate your relationships with faculty members? (Please use a 
scale from 1 (unavailable and unhelpful) to 7 (available and helpful) with four 
being right in the middle of the two extremes). 

(b) Overall, how would you evaluate the quality of academic advising you have 
received at UMaine? (Please use excellent, good, fair, or poor). 

(c) How would you evaluate your entire educational experience at UMaine? 
(Please use excellent, good, fair, or poor). 

(d) If you could start over again, would you still go to UMaine? (Please use 
definitely yes, probably yes, probably no, or definitely no). 

We chose these questions because they provide a view of a student’s overall level of satisfaction 
and perceived relationships with faculty members — and without creating a complicated 
interaction between interviewer and interviewee. The questions are each self-contained (i.e., 
they do not require interviewees to use short-term memory to refer back to a previous question) 
and straightforward, and they offer simple response options. (The interview script is available 
upon request.) 

In the analysis phase of the project, we compared the responses to the four questions 
above with the responses provided by UMaine NSSE participants. We used regression analysis 
to examine if there was a statistically significant difference between responders and the 
telephone interviewees in student-faculty relationship ratings. Because NSSE, in its standard 
reporting, takes into account level, gender, and enrollment status (either through a separation of 
results or a weighting procedure), we included these characteristics as control variables. 
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Specifically, we regressed the student-faculty rating on a nonresponder indicator variable (0 = 
NSSE responder, 1 = telephone interviewee), level (0 = freshman, 1 = senior), gender (0 = male, 
1 = female), and enrollment status (0 = full-time, 1 = part-time). 

To determine if responders and telephone interviewees differed in their overall 
satisfaction with their UMaine experience, we first collapsed categories to create dichotomous 
variables from the responses to each of the three overall satisfaction questions. We transformed 
the overall advising and overall educational experience responses to good/excellent or fair/poor, 
and the likelihood of returning to UMaine response to definitely yes/probably yes or probably 
no/definitely no. We then used logistic regression, a methodology which allows for a 
dichotomous dependent variable, to regress the transformed responses on, as before, the 
nonresponder indicator variable (0 = NSSE responder, 1 = telephone interviewee), level (0 = 
freshman, 1 = senior), gender (0 = male, 1 = female), and enrollment status (0 = full-time, 1 = 
part-time). 

Results 

Demographic Comparisons 

Table 1 displays the overall and class-level response rates. Overall, approximately 21% 
of the 4,178 students who received invitations to complete the NSSE responded. There was a 
statistically significant difference in response rates between freshmen and seniors ip < .01): The 
freshman response rate was 19% compared with 24% among seniors. Such a difference is not 
unusual, however: NSSE reported response rates of approximately 25% and 28%, respectively 


(NSSE, 2011). 
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Table 1. Response Rates by Class 



Freshmen 

Seniors 

Total 

Total Surveys Administered 

2,057 

2,121 

4,178 

Respondents 

391 

502 

893 

Non-Respondents 

1,666 

1,619 

3,285 

UMaine Response rates 

19% 

24% 

21% 

Overall NSSE response rates 

25% 

28% 

27% 


Note. Adapted from UMaine Institutional Report 2011 (Respondent Characteristics), 
National Survey of Student Engagement, 2011. 


Tables 2 and 3 provide freshman and senior response rates broken down by gender, 
residency, enrollment status, transfer status, cumulative GPA, and college of major. ' The tables 
show the results of the x tests of independence, which assess whether there were differences 
across groups in the proportion of students who responded to the survey, representing an over- or 
under-representation among respondents. For example, the results in Table 2 show a statistically 
significant difference related to gender. One can see that females had a higher response rate 
(26.5%) than males (15.3%) and, further, were more highly represented among the respondents 
(61.4% vs. 38.6%). This distribution of males and females is in contrast to that in the total 
student population (47.9% vs. 52.1%). If there was no relationship between gender and 
response, the split between females and males among the respondents would have been more 
reflective of that seen in the total population. 

Among freshmen, a significantly higher proportion of females, first-time students, and 
students living on campus responded to the survey. As for seniors, more likely to respond were 

* We created our dataset from the spring database, which only includes students who are registered for courses 
during the spring semester. Because the names supplied to NSSE were taken from the fall 2010 student database, 
some students who received surveys were not registered for classes in spring 2011. To restrict the sample to only 
students who were enrolled during the time of the survey administration, we excluded these students. This resulted 
in the exclusion of six freshmen and 13 seniors who responded to the NSSE survey. 



14 


females, full-time students, nonresidents, and students living on campus. Academic 
achievement, as measured by cumulative GPA, also was related to survey participation: For both 
freshmen and seniors, the response rate of students with GPAs above 3.5 was roughly double 
that of students with GPAs below 2.5. Across colleges, there were differences in response rates 
among seniors but not among freshmen. The College of Natural Sciences, Forestry, and 
Agriculture enjoyed the highest senior response rate (32.5%), while the College of Business, 
Public Policy, and Health showed the lowest (18.6%). 


^ BPPH = College of Business, Public Policy, and Health, EHD = College of Education and Human Development, 
ENGR = College of Engineering, LAS = College of Liberal Arts and Sciences, NSEA = College of Natural 
Sciences, Eorestry, and Agriculture. Eurther, DLL = Division of Lifelong Learning and EXPL = Explorations (a 
program for undecided students and/or students requiring additional academic preparation). 
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Table 2. Demographic Comparisons: Freshmen 


Demographic Groups 

UMaine 

Population 

NSSE 

Respondents 

Response 
; Rate 

% of 

Population 

% of 

Respondents 

Test of 
Independence 

/ df P 

Gender 









Females 

896 

237 

26.5% 

47.9% 

61.4% 




Males 

976 

149 

15.3% 

52.1% 

38.6% 

35.7 

1 

<.01 

Enrollment status 









Full-time 

1,789 

374 

20.9% 

95.6% 

96.9% 




Part-time 

83 

12 

14.5% 

4.4% 

3.1% 

2.0 

1 

.16 

Transfer status 









New 

1,783 

376 

21.1% 

95.4% 

97.4% 




Transfer 

86 

10 

11.6% 

4.6% 

2.6% 

4.5 

1 

<.05 

Residency 









Resident 

1,501 

306 

20.4% 

80.2% 

79.3% 




Nonresident 

371 

80 

21.6% 

19.8% 

20.7% 

.3 

1 

.62 

Living situation 









On-campus 

1,513 

332 

21.9% 

80.2% 

86.0% 




Off-campus 

359 

54 

15.4% 

19.2% 

14.0% 

8.4 

1 

<.01 

College 









BPPH 

134 

25 

18.7% 

7.2% 

6.5% 




EHD 

157 

26 

16.6% 

8.4% 

6.7% 




ENGR 

325 

61 

18.8% 

17.4% 

15.8% 




NSFA 

428 

99 

23.1% 

22.9% 

25.7% 




LAS 

558 

131 

23.5% 

29.8% 

33.9% 




(EXPL) 

223 

33 

14.8% 

11.9% 

8.6% 




(DLL) 

47 

11 

23.4% 

2.5% 

2.9% 

11.9 

6 

.07 

GPA 









Below 2.5 

685 

108 

15.8% 

37.0% 

28.1% 




2.5 - 2.99 

407 

64 

15.7% 

22.0% 

16.7% 




3.0 - 3.49 

408 

93 

22.8% 

22.0% 

24.2% 




3.5 or higher 

352 

119 

33.8% 

19.4% 

31.7% 

54.2 

3 

<.01 
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Table 3. Demographic Comparisons: Seniors 


Demographic Groups 

UMaine 

Population 

NSSE 

Respondents 

Response 

Rate 

% of 

Population 

% of 

Respondents 

Test of 
Independence 

/ df P 

Gender 









Females 

935 

267 

28.6% 

46.5% 

54.0% 




Males 

1,075 

227 

21.1% 

53.5% 

46.0% 

14.9 

1 

<.01 

Enrollment status 









Full-time 

1,570 

408 

26.0% 

78.1% 

82.6% 




Part-time 

440 

86 

19.6% 

21.9% 

17.4% 

7.7 

1 

<.01 

Transfer status 









New 

1,405 

353 

25.1% 

70.2% 

71.8% 




Transfer 

596 

139 

23.3% 

29.8% 

28.2% 

.7 

1 

.40 

Residency 









Resident 

1,758 

420 

23.9% 

87.5% 

85.0% 




Nonresident 

252 

74 

29.4% 

12.5% 

15.0% 

3.6 

1 

.06 

Living situation 









On-campus 

282 

101 

35.8% 

14.0% 

20.5% 




Off-campus 

1,728 

393 

22.7% 

86.0% 

79.5% 

26.1 

1 

<.01 

College 









BPPH 

247 

46 

18.6% 

12.3% 

9.3% 




EHD 

225 

56 

24.9% 

11.2% 

11.3% 




ENGR 

404 

89 

22.0% 

20.1% 

18.0% 




NSFA 

502 

163 

32.5% 

25.0% 

33.0% 




LAS 

608 

138 

22.7% 

30.2% 

28.0% 




(DLL) 

24 

2 

8.3% 

1.2% 

0.4% 

27.6 

5 

<.01 

GPA 









Below 2.5 

229 

29 

12.7% 

11.4% 

5.9% 




2.5 - 2.99 

540 

107 

19.8% 

26.9% 

21.7% 




3.0 - 3.49 

724 

195 

26.9% 

36.1% 

39.5% 




3.5 or higher 

515 

163 

31.7% 

26.4% 

33.5% 

40.2 

3 

<.01 
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2 

In summary, the results of the x tests of independence show statistically significant 
differences across demographic groups. An overview of the groups with disproportionately high 
response rates appears in Table 4. 


Table 4. Summary of Demographic Comparisons 


More likely to respond among freshmen were: 

• Females 

• New students (as opposed to transfers) 

• Students living on campus 

• Students with higher GPAs 


More likely to respond among seniors were: 

• Females 

• Full-time students 

• Students living on campus 

• Students in the college of NSFA 

• Students with higher GPAs 


Although there were differences in response rates across demographic groups, such 
differences are only indicative of nonresponse bias if there were also discrepancies in responses 
to pertinent survey items. To identify whether such a threat exists, we conducted between-group 
comparisons using the demographic categories in Table 4 on the four items used in the 
nonresponder analysis (relationships with faculty members, quality of academic advising, 
perception of overall experience, and likelihood of attending again) and the five benchmark 
scores (Academic Challenge, Active and Collaborative Learning, Student-Faculty Interaction, 
Enriching Educational Experiences, Supportive Campus Environment). We used the % test of 
independence for the ordinal and nominal items and the independent sample t test or one-way 
analysis of variance for those with interval responses. The following is a summary of the 


statistically significant findings, with the complete results available upon request. 
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Among freshmen, there was a statistically significant difference associated with living 
situation: Freshmen living off campus had a more positive view of their interaction with faculty 
members (M = 35.8, SD = 20.5) than those living on campus (M = 29.3, SD = 16.8), 1(339) = 
2.40, p < .05. This difference corresponded to an effect size of approximately one third of a 
standard deviation (d = .35). Also, there was a significant effect associated with GPA and the 
Academic Challenge benchmark (F(3, 329) = 2.81, p < .05). However, follow-up pairwise 
comparisons were not statistically significant. Such a contradictory finding can be attributed to 
the conservative nature of the pairwise comparison test which, in order to reduce the probability 
of Type 1 error, corrects for the multiple comparisons being made. 

The threat of nonresponse bias appears to be more pronounced among seniors, with 
discrepancies seen in relation to gender, college, and GPA. Males and females differed 
significantly in perceptions of their overall educational experience N = 423) = 5.61, p < 
.01): 85% of females indicated their overall educational experience was good or excellent 
compared with 76% of males. In addition to being more satisfied with their overall experience, 
senior females also showed slightly higher Academic Challenge benchmark scores than males 
(1(446) = 3.0, p < .01), corresponding to an effect size of roughly one quarter of a standard 
deviation {d = .28). 

Seniors of varying achievement levels differed in their perceptions of their relationships 
with faculty members {F(2, 434) = 4.5, p < .01) and the support provided by the campus 
environment (F(3, 417) = 3.4, p < .05). Tukey post-hoc comparisons indicate that students with 
GPAs of 3.5 or higher reported significantly more favorable sentiments regarding interactions 
with faculty (M = 5.5, SD = 1.2) than students with GPAs below 2.5 (M = 4.6, SD = 1.6), 
equaling an effect size of .30. Students with GPAs of 3.5 or higher also scored more highly on 
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the Supportive Campus Environment benchmark (M = 56.6, SD = 18.0) than students with GPAs 
of 2.5 or below (M = 43.6, SD = 23.5), corresponding to an effect size of .30. 

Finally, seniors across colleges differed in their perceptions of the quality of academic 
advising they received (z^(4) = 11.5, p < .05) and the support provided by the campus 
environment (F(4, 416) = 5.0, p < .01). ENGR seniors were the most positive in regard to 
academic advising (75% indicating good or excellent), while their EAS counterparts were the 
least positive (51% reporting good or excellent). Students in EHD were highest on the 
Supportive Campus Environment benchmark (M = 61.0, SD = 16.9), whereas students in NSFA 
(M= 51.4, SD = 18.6) and EAS (M= 51.8, SD = 18.4) were the lowest. The differences between 
the EHD students and those in NSFA and EAS correspond to effect sizes of .26 and .25, 
respectively. 

Follow-up telephone interviews among nonresponders 

Response rates. Table 5 displays a summary of the number of calls attempted, and the 
number and percentage of students who opted to participate in the telephone interviews. 
Approximately one third of the students with valid phone numbers for whom contact attempts 
were made participated in the interviews. Combined, almost 80% of the freshmen and seniors 
we ultimately were able to reach agreed to participate in the telephone interview. The telephone 
interviewees represent approximately 3% of the nonresponder population as a whole. 



20 


Table 5. Response Rates for Follow-Up Analysis 



Freshmen 

Seniors 

Total phone numbers attempted 

193 

175 

Incorrect or disconnected numbers 

41 

31 

Total valid numbers 

152 

144 

Declined to be interviewed 

11 

15 

Total telephone survey participants 

50 

50 

Total not reached 

91 

79 

Telephone survey participants as percentage of 
students reached 

82% 

77% 

Telephone survey participants as percentage of 
valid numbers 

33% 

35% 

Telephone survey participants as percentage of 
all nonresponders 

3% 

3% 


Representativeness of telephone sample. Tables 6 and 7 report the demographic 
characteristics of the telephone interviewees compared with the characteristics of the 
nonresponder population as a whole. We used x goodness of fit tests to examine whether the 
differences between the two groups were statistically significant. The results show that the 
sample of interviewees was generally representative of the population of nonresponders. The 
one area where the nonresponder and telephone survey samples differed is in the proportion of 
freshmen in the various colleges: EXPL students were more highly represented among 
telephone interviewees than nonresponders (28% vs. 12.8%), while the opposite was true for 


BPPH (2% vs. 7.3%) and ENGR (12% vs. 17.8%) students. 
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Table 6. Nonresponder Population vs. Telephone Survey Interviewees: Demographic 
Characteristics of Freshmen 


Nonresponder Telephone ^ Goodness of 

Population Interviewees Fit Test 


Demographic 

Groups 

n 

% 

n 

% 


df 

P 

Gender 








Females 

659 

44.3% 

23 

46.0% 




Males 

827 

55.7% 

27 

54.0% 

.06 

1 

.81 

Enrollment Status 








Full-time 

1,415 

95.2% 

47 

94.0% 




Part-time 

71 

4.8% 

3 

6.0% 

.16 

1 

.69 

Transfer Status 








First- year 

1,407 

94.9% 

49 

98.0% 




Transfer 

76 

5.1% 

1 

2.0% 

.99 

1 

.32 

Residency 








In- state 

1,195 

80.4% 

42 

84.0% 




Out-of-state 

291 

19.6% 

8 

16.0% 

.41 

1 

.52 

Living Situation 








On-campus 

1,181 

79.5% 

35 

70.0% 




Off-campus 

305 

20.5% 

15 

30.0% 

2.77 

1 

.10 

College 








BPPH 

109 

7.3% 

1 

2.0% 




EHD 

131 

8.8% 

3 

6.0% 




ENGR 

264 

17.8% 

6 

12.0% 




NSEA 

329 

22.1% 

11 

22.0% 




LAS 

427 

28.7% 

15 

30.0% 




(EXPL) 

190 

12.8% 

14 

28.0% 




(DLL) 

36 

2.4% 

0 

0.0% 

12.03 

5 

.03 

GPA 








Below 2.5 

566 

39.4% 

24 

48.0% 




2.5 - 2.99 

332 

23.1% 

14 

28.0% 




3.0 - 3.49 

307 

21.3% 

4 

8.0% 




3.5 or higher 

233 

16.2% 

8 

16.0% 

5.61 

3 

.13 
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Table 7. Nonresponder Population vs. Telephone Survey Interviewees: Demographic 
Characteristics of Seniors 


2 

Nonresponder Telephone % Goodness of 

Population Interviewees Fit Test 


Demographic Groups 

n 

% 

n 

% 


df 

P 

Gender 








Females 

668 

44.1% 

25 

50.0% 




Males 

848 

55.9% 

25 

50.0% 

.71 

1 

.40 

Enrollment Status 








Full-time 

1,162 

76.6% 

37 

74.0% 




Part-time 

354 

23.4% 

13 

16.0% 

.19 

1 

.66 

Transfer Status 








First- year 

1,052 

69.7% 

32 

64.0% 




Transfer 

457 

30.3% 

18 

36.0% 

.77 

1 

.38 

Residency 








In- state 

1,338 

88.3% 

42 

84.0% 




Out-of-state 

178 

11.7% 

8 

16.0% 

.90 

1 

.34 

Living Situation 








On-campus 

181 

11.9% 

5 

10.0% 




Off-campus 

1,335 

88.1% 

45 

90.0% 

.17 

1 

.68 

College 








BPPH 

201 

13.3% 

5 

10.0% 




EHD 

169 

11.1% 

5 

10.0% 




ENGR 

315 

20.8% 

8 

16.0% 




NSEA 

339 

22.4% 

13 

26.0% 




LAS 

470 

31.0% 

19 

38.0% 




(DLL) 

22 

1.5% 


0% 

2.06 

4 

.73 

GPA 








Below 2.5 

181 

12.3% 

7 

14.0% 




2.5 - 2.99 

414 

28.2% 

13 

26.0% 




3.0 - 3.49 

520 

34.4% 

16 

32.0% 




3.5 or higher 

352 

24.0% 

14 

28.0% 

.70 

3 

.87 
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Reasons for nonresponse. When asked the reason for their non-response to the NSSE, 
those students who did remember receiving the survey — approximately two thirds — offered two 
primary reasons: they were too busy, or they just did not get to it or feel like it. There were only 
minimal differences between freshmen and seniors in response to this question. Table 8 shows 
the breakdown by class level. 


Tables. Reasons for Nonresponse 


Freshmen Seniors 


n 

% 

n 

% 


Don't remember the survey 

19 

38.0% 

18 

36.7% 

Was too busy 

14 

28.0% 

13 

26.5% 

Just did not get to it or feel like it 

17 

34.0% 

16 

32.7% 

Other 

0 

0.0% 

2 

4.1% 


Note. One senior did not provide a reason for nonresponse. 


Relationships with faculty memhers. Overall, telephone interviewees indicated a 
slightly more favorable perception of their relationships with faculty members than did NSSE 
responders. Table 9 displays the mean ratings for responders and telephone interviewees. 
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Table 9. Responders vs. Telephone Interviewees: 
Relationships with Faculty Members 


Telephone 


Responders 

Interviewees 

n M SD 

n M SD 


Overall 

761 

5.2 

1.4 

96 

5.5 

1.1 

Eevel 

Ereshmen 

323 

5.0 

1.4 

48 

5.3 

1.2 

Seniors 

438 

5.3 

1.4 

48 

5.7 

1 

Gender 

Males 

324 

5.2 

1.4 

50 

5.4 

1.0 

Eemales 

437 

5.1 

1.4 

46 

5.6 

1.2 

Enrollment status 

Eull-time 

674 

5.1 

1.4 

80 

5.4 

1.1 

Part-time 

87 

5.3 

1.3 

16 

5.9 

1.1 


Note. Four telephone interviewees did not respond to the question. 


To determine if there was a statistically significant difference between responders and 
telephone interviewees, we regressed the faculty relationship rating on an indicator variable 
distinguishing telephone interviewees from the NSSE survey responders (0 = NSSE survey 
responders, 1 = telephone interviewees,). We included gender (0 = male, 1 = female), class level 
(0 = freshman, 1 = senior), and enrollment status (0 = full-time, 1 = part-time) as control 
variables. The results, which appear in Table 10, indicate that overall the model is statistically 
significant (E(4,852)=3.27,p < .05) and, further, there is a statistically significant difference in 
ratings between responders and telephone interviewees (t = 2.31, p < .05). Telephone 
interviewees reported ratings that were approximately one third of a point higher than responders 
(based on a seven-point scale). Class level also was a statistically significant predictor {t = 2.43, 
p < .01), with seniors showing ratings approximately one quarter of a point higher than those of 
freshmen. Neither gender nor enrollment status were statistically significant predictors. 
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Although the response and class-level indicator variables were both statistically significant, the 
magnitude of their effects is rather small. The R shows that the model only explains 
approximately 1.5% of the variance in the ratings. 


Table 10. Predicting Student-Faculty Relationship Ratings 



Coefficients 

Overall Model 


b 

t 

R^ F 

Constant 

4.88** 

27.60 


Telephone interviewee 
indicator 

.34* 

2.31 


Class level 

24 ** 

2.43 


Gender 

-.05 

-.56 


Enrollment status 

.09 

.58 

.015 3.27 

* p < .05 


** p < .01 


Overall satisfaction. Tables 11-12 display student responses to questions about the 
quality of academic advising received and their overall educational experience; Table 13 shows 
the responses when asked whether or not they would still attend UMaine if they had the chance 
to start over. For ease of interpretation, we reduced the four-point scales to dichotomous 
variables: excellent/good vs. fair/poor for academic advising and overall experience, and 
definitely yes/probably yes vs. probably no/definitely no for likelihood of attending UMaine 
again. 

There was little difference between responders and telephone interviewees in their 
perceptions of the overall quality of academic advising they received (75% vs. 72% for freshmen 
and 62% vs. 61% for seniors). In contrast, telephone interviewees were more likely than 
responders to rate their overall educational experience as good or excellent (96% vs. 85% for 
freshmen and 88% vs. 81% for seniors), and to indicate that they would attend UMaine again if 
they had the chance to start over (90% vs. 82% for freshmen and 96% vs. 75% for seniors). 
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Table 11. Responders vs. Telephone Interviewees: Quality of Academic Advising 


Responders Telephone Interviewees 


Poor/Fair 

Good/Excellent 

Poor/Fair 

Good/Excellent 

n % 

n % 

n % 

n % 


Overall 

233 

32.2% 

490 

67.8% 

32 

33.3% 

64 

66.7% 

Eevel 

Ereshmen 

76 

24.9% 

229 

75.1% 

13 

27.7% 

34 

72.3% 

Seniors 

157 

37.6% 

261 

62.4% 

19 

38.8% 

30 

61.2% 

Gender 

Males 

99 

32.5% 

206 

67.5% 

20 

40.0% 

30 

60.0% 

Females 

134 

32.1% 

284 

67.9% 

12 

26.1% 

34 

73.9% 

Enrollment 

status 

Full-time 

207 

32.4% 

431 

67.6% 

28 

35.0% 

52 

65.0% 

Part-time 

26 

30.6% 

59 

69.4% 

4 

25.0% 

12 

75.0% 


Note. Four telephone interviewees did not respond to the question. 
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Table 12. Responders vs. Telephone Interviewees: Overall Educational Experience 

Responders Telephone Interviewees 

Poor/Fair Good/Excellent Poor/Fair Good/Excellent 



n 

% 

n 

% 

n 

% 

n 

% 

Overall 

124 

17.2% 

599 

82.8% 

8 

8.1% 

91 

91.9% 

Eevel 

Freshmen 

45 

14.8% 

260 

85.2% 

2 

4.1% 

47 

95.9% 

Seniors 

79 

18.9% 

339 

81.1% 

6 

12.0% 

44 

88.0% 

Gender 

Males 

65 

21.3% 

240 

78.7% 

4 

7.8% 

47 

92.2% 

Females 

59 

14.1% 

359 

85.9% 

4 

8.3% 

44 

91.7% 

Enrollment status 

Full-time 

106 

16.6% 

532 

83.4% 

8 

9.6% 

75 

90.4% 

Part-time 

18 

21.2% 

67 

78.8% 

0 

0.0% 

16 

100.0% 


Note. One telephone interviewee did not respond to the question. 


Table 13. Responders vs. Telephone Interviewees: Likelihood of Attending Again 


Responders Telephone Interviewees 

Definitely Probably Definitely Probably 

no/Probably yes/Definitely no/Probably yes/Definitely 


no 

yes 


no 


yes 

n % 

n % 

n 

% 

n 

% 


Overall 

160 

22.1% 

565 

77.9% 

7 

7.1% 

92 

92.9% 

Eevel 

Freshmen 

54 

17.6% 

252 

82.4% 

5 

10.2% 

44 

89.8% 

Seniors 

106 

25.3% 

313 

74.7% 

2 

4.0% 

48 

96.0% 

Gender 

Males 

69 

22.6% 

236 

77.4% 

3 

5.9% 

48 

94.1% 

Females 

91 

21.7% 

329 

78.3% 

4 

8.3% 

44 

91.7% 

Enrollment 

status 

Full-time 

139 

21.7% 

501 

78.3% 

7 

8.4% 

76 

91.6% 

Part-time 

21 

24.7% 

64 

75.3% 

0 

0.0% 

16 

100.0% 


Note. One telephone interviewee did not respond to the question. 
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To identify whether there was a statistically difference between responders and telephone 
interviewees in how they responded to these three questions, we conducted three separate logistic 
regression analyses in which we regressed each of the variables described in Tables 1 1-13 on the 
nonresponder indicator variable, gender, class level, and enrollment-status indicators. The 
results, which appear in Table 14, show no statistically significant difference between responders 
and telephone interviewees in their perceptions of academic advising. However, telephone 
interviewees were more likely than responders to indicate their overall educational experience 
was positive = 5.27, p < .05) and that they would attend UMaine if given the chance to do it 
again (x = 10.02, p < .01). Enrollment status was not a significant predictor in either model, but 
females were more likely than males to positively rate their overall educational experience (x = 
5.26,p< .05). 

The odds ratio is helpful in interpreting the magnitude of these statistically significant 
effects. The odds of the telephone interviewees indicating they had a positive overall experience 
(i.e., good or excellent) was almost two and a half times that of the NSSE responders, holding 
level, gender, and enrollment status constant. The odds ratio was higher when it came to 
indicating whether they would return to UMaine if they had to do it again. Here, the odds of the 
telephone interviewees stating they would return if given another chance was roughly three and a 
half times that of the NSSE survey responders. 
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Table 14. Predicting Overall Satisfaction Items 


Overall Educational 

Eikelihood of Attending 

Experience 

Again 

„ 2 Odds 

„ 2 Odds 

^ Ratio 

^ Ratio 


Constant 

.78 

3.23 

2.18 

-.22 

0.25 

0.80 

Telephone interviewees 

.88* 

5.27 

2.41 

1.28** 

10.02 

3.59 

Class level 

.32 

2.39 

1.37 

-.50* 

7.03 

0.61 

Gender 

.44* 

5.26 

1.55 

.00 

0.00 

1.00 

Enrollment status 

-.05 

0.03 

0.95 

.09 

0.11 

1.09 

* p < .05 

** p < .01 


Discussion 

Overview of Findings 

The purpose of this study was to identify whether there is evidence of nonresponder bias 
in the NSSE data collected from UMaine students in spring 2011. Specifically, we addressed the 
questions of whether nonresponders and responders were demographic ally similar, why 
nonresponders did not participate in the survey, and how responders and nonresponders may 
have differed in their perceptions of UMaine. 

The demographic analysis suggests NSSE respondents did differ on such key 
characteristics as gender, enrollment status, living situation, GPA, and college. Eemales, 
students living on campus, and students with higher GPAs were among the freshmen most likely 
to respond; and females, full-time students, those living on campus, majors in NSEA, and 
students with higher GPAs were among the seniors most likely to respond. Many of these 
differences are consistent with what has been reported in survey research specific to higher 
education. Males, part-time students, and students with lower GPAs typically are less likely to 
respond to surveys than their counterparts (Kuh et ah, 2001; Porter & Umbach, 2006; Sax, 
Gilmartin & Bryant, 2003). Although the differences across colleges may be partially related to 
a general propensity among students in certain programs to respond to surveys, these differences 
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also likely reflect differences among colleges in efforts aimed at bolstering student response to 
the survey. 

The results of the demographic analyses of responders reveals that, in some cases, groups 
more highly represented among responders may have had differing perceptions from those 
underrepresented. Among freshmen, specifically, there were differences associated with living 
situation; among seniors, there were differences among gender, GPA, and college. For example, 
seniors with GPAs above 3.5 (who were more highly represented among responders than those 
with lower GPAs) were also more positive about their interactions with faculty members and the 
level of support provided by the campus environment. Although there were discrepancies among 
disproportionately represented groups, which suggest some degree of nonresponse bias, the 
effects are rather small, with d’s ranging from .25 to .35. 

As mentioned, one of the purposes of the follow-up analysis was to gain insight into why 
students did not respond to the survey. The results of the telephone interviews reveal two key 
findings in this regard. First, the majority of interviewees confirmed they did in fact receive the 
email invitation to participate in the survey. Second, interviewees offered a lack of time or 
motivation as their primary reasons for nonresponse. Although it is possible that either of these 
reasons could correlate with student engagement, there was no evidence to suggest that students 
consciously did not respond because of their dissatisfaction with UMaine. 

With the follow-up analysis, we sought to identify whether responders and nonresponders 
differed in their perceptions of UMaine. To keep interviews simple and practical, we only 
included four overall measures of satisfaction rather than items specifically related to student 
behaviors. Although this limitation restricts one’s ability to draw conclusions about levels of 
engagement, the questions do provide a view of students’ level of satisfaction with their faculty 
interactions, the quality of advising they received, and the overall experience in general. These 
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are important considerations in their own right. And when combined with the question regarding 
a student’s likelihood of returning to UMaine given a second chance, they can discriminate 
between students who had positive UMaine experiences and those having had less than positive 
experiences. 

The results of the follow-up analysis suggest that, with the exception of academic 
advising, telephone interviewees had a more favorable view of their overall experience than 
responders. This should be interpreted with caution for two reasons. First, the use of telephone 
interviews is likely (and ironically) to introduce bias: Students may have a greater tendency to 
provide favorable answers on the phone than in a paper or online survey (Dillman et. ah, 1996). 
In this respect, it is not surprising that the present results indicate more positive views among 
telephone interviewees. Second, the magnitude of some differences between the groups was 
quite small. For example, statistical significance notwithstanding, the telephone interviewees 
reported faculty interaction ratings that were only one third of a point (on a seven-point scale) 
higher than those of the responders. 

These limitations, however, should not be interpreted as meaning that the telephone 
survey results are inflated and carry little import. If students were willing to express on the 
phone their discontent with their academic advising experience, they arguably would have been 
equally willing to disclose an unequivocal dissatisfaction with their UMaine experience. Erring 
on the side of caution, then, perhaps the best conclusion is by way of what these data do not 
suggest: They do not indicate that nonresponders are any less satisfied with their UMaine 
experience than responders. 

Implications 

Three major conclusions surface from our analyses: (a) there is evidence to suggest that 
NSSE responders are not demographically representative of the student population, (b) 
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nonresponse among students was primarily due to their lack of time or motivation, and (c) the 
evidence does not indicate that NSSE responders as a whole were more satisfied with their 
UMaine experience than nonresponders. Each of these conclusions has implications that will 
help UMaine better understand the NSSE results and their use. 

Demographic differences. There were significant differences in response rates across 
demographic groups. Eortunately, the differences associated with gender and enrollment status 
are accounted for by a weighting procedure in NSSE’s standard reporting. However, NSSE does 
not account for differences associated with living situation, GPA, and college of major. The 
discrepancies in the perceptions of students across groups within these demographic categories, 
although modest, are an issue that should be considered when interpreting the results. The 
campus-wide results will be more heavily impacted by the perceptions of overrepresented groups 
than those of the underrepresented, which could produce results that are more or less favorable 
than would have been found if respondents were demographically similar to the student 
population. 

Reason for Response. The UMaine community can find comfort in the fact that there 
did not appear to be a problem with the process through which the survey invitations were sent 
out and, further, that students did not indicate their lack of response was due to strong negative 
feelings toward UMaine or the survey itself. In most cases, nonresponse was a function of a lack 
of time or general forgetfulness. 

Nonresponse bias. A concern about the NSSE, and other surveys of a similar nature, is 
that students who respond may be more engaged in their college experience than those who did 
not (Kuh, 2003; Porter & Umbach, 2006; Porter & Whitcomb, 2005 ). The findings from the 
follow-up study fail to support such a concern, revealing no evidence to suggest responders were 
more satisfied with their experience than nonresponders. Eurther contradicting the presumption 
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that responders are more engaged, the demographic analyses revealed that students in 
underrepresented groups did not indicate consistently less favorable perceptions than students in 
overrepresented groups. For some groups on some items, it was the students more highly 
represented who were the most positive, but in other cases it was the underrepresented groups 
that responded more favorably. 

Taken together, the findings from the demographic and follow-up analyses hold at least 
two implications regarding nonresponse bias. First, although there may be some degree of 
nonresponse bias, the results are not necessarily an inflated view of student perceptions. Second, 
any bias appears to be more a function of response rate differences across demographic groups 
than a fundamental difference between responders and nonresponders in general. The latter, 
coupled with the fact that the differences between overrepresented and underrepresented groups 
are small, suggests that disaggregating the results by demographic categories may help reduce 
the potential negative impact of nonresponse bias on the interpretation of the results. Had the 
follow-up study shown the perceptions of the nonresponders in general differed greatly from 
those of the responders, there would be little potential for minimizing the bias because the views 
of the nonresponders would remain unknown. 
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Concluding thoughts 

The NSSE provides UMaine with meaningful feedback on student perceptions of their 
UMaine experience, and the results of the follow-up interviews suggest there is no reason to 
believe that responders as a whole are any more satisfied with their experience than 
nonresponders. However, some caution is advised when interpreting the results due to varying 
response rates and student perceptions across key demographic groups. Although the differences 
identified in this study are small and not likely to have a marked impact on the overall 
conclusions, they do point to the need for UMaine to be cautious when interpreting the campus 
wide results, and, in particular, to disaggregate the data beyond that provided in the standard 
NSSE reports. In our view, the UMaine results should be disaggregated by GPA, living 
situation, and college — the three areas where there were both response rate and perceptual 


differences in our data. 
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Abstract 

An alumni survey designed to assess institutional effectiveness, identify internship & career 
opportunities for current students, and obtain information to support fundraising and recruitment 
was implemented at Regis College. The ways in which the results were used are detailed, 
followed by a discussion of lessons learned and plans for future research and actions. 
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Maximizing the Utility of Alumni Feedback 


Assessing institutional effectiveness is a “complicated collage of evaluation and 
assessment challenges for institutions” (Volkwein, 2010a, p. 26). Alumni surveys are a powerful, 
but sometimes underutilized assessment tool. Alumni can provide a unique perspective of 
institutional effectiveness that is more complete than that of current students or any other 
constituency group. Alumni have direct experience with both an institution and with the impact 
of their education once they graduate. As Volkwein points out (2010b), the alumni perspective 
and connection is increasingly important to institutions of higher learning because alumni are 
now recognized as important sources of both information and financial support. Maintaining an 
accurate database of contact information, reaching alumni in an effective and efficient manner, 
and getting alumni to respond to requests for information all present challenges. An alumni 
survey was implemented at Regis College despite those challenges. The purpose of this was to 
improve our database of alumni information, identify internship and career opportunities for 
current students, obtain information concerning institutional effectiveness to prepare for a 
reaccreditation review, and to obtain information that would support future fundraising and 
recruitment efforts. As Volkwein succintly states, “Alumni are important sources of information 
and support, and alumni studies should occupy a prominent place in the institutional research 
portfolio”(2010b, p. 137). 

Alumni surveys have been used for a variety of purposes and can have a significant 
impact on public policy and advancement of higher education. Cabrera, Weerts, and Zulick 
(2005) discuss the three primary purposes of alumni surveys: 1) outcomes data, 2 engagement 
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and competencies information, and 3) alumni giving. Previous literature indicates that inclination 
to give can be predicted by the quality of an individual’s educational experience, the extent to 
which an institution prepared the graduate for a career, the extent to which a graduate maintains 
contact with the institution, the graduate’s current impressions of the institution, and the 
graduate’s capacity to give (Cabrera, Weerts, and Zulick, 2005). All of these factors can be 
assessed with an alumni survey. This knowledge can then be used for targeted fundraising 
appeals. 

Alumni surveys are typically implemented for a rather limited purpose by a single 
constituency group (Borden, 2005). For example, Davidson-Shivers, Inpornjivit & Sellers (2004) 
surveyed 125 alumni of an instructional design master’s and doctoral program (along with 100 
current students) in order to evaluate the programs and facilitate strategic planning. Landrum and 
Lisenbe (2008) also used an alumni survey to evaluate instructional and departmental quality. 
This information was used for both shaping the direction of an academic department, and to 
inform students about opportunities available to them after graduation. 

Bossart, Wentz and Heller (2009) described the results of an alumni survey administered 
at the University of Wisconson-Strout. Consistent with previous research, their findings 
indicated that alumni were very satisfied with their program, would attend UW-Strout again if 
they could do it over, and felt that their education was worth their investment in time and money. 
In 2008 Hennessy (as cited in Bossart, Wentz and Heller, 2009) reported that 78% of college 
graduates would attend the same university again if they could do it over. The UW-Strout results 
also showed that career advising and academic advising earned the lowest alumni ratings, which 
is consistent with previous surveys, such as the survey at Southwest Texas State University 


(Ogletree, 1999). Career and academic advising are areas that are increasingly important. 
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College students, and their families, are focused on the “value proposition” of a college 
education more strongly than ever before, which leads to an increased desire for explicit 
outcomes data and career services as they make their college decisions. Alumni research can 
provide important information to help meet these demands. 

Borden (2005) points out that it is both possible and desirable for campuses to align 
alumni surveys to serve multiple purposes, rather than for a limited purpose by a single 
constituency group. It is the role of the institutional researcher to implement such a survey, and 
to ensure that the results of the survey are interpreted, disseminated and applied to policy 
decisions. Borden states that effective use of alumni survey results are “pivotal to linking 
improvement and accountability” (Borden, 2005, p.70). This position is consistent with the 
recommendations made by Volkwein (2010a & 20010b). In addition to a multifunctional survey, 
Davidson-Shivers, Inpornjivit & Sellers (2004) state that for data from alumni to continue to be 
useful, the databases must be maintained and feedback updated. Hoey & Gardner (1999) report 
results of an ongoing alumni and employer survey program used to improve institutional 
effectiveness. They point out that the maximum effectiveness of this approach does not begin to 
accrue until several cycles of data collection occur. At least three cycles of data collection are 
needed for trend analysis to begin. 

This paper describes a multifunctional alumni survey which was implemented at Regis 
College. The survey was intended to meet the needs of our career planning, academic planning, 
institutional advancement and enrollment offices; to help us assess institutional effectiveness; 


and to provide baseline data that would enable us to analyze institutional trends going forward. 
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Method 

A survey was made available to all Regis College Alumni in electronic and paper 
formats: through a link on our alumnae page, in an email to all alumni who have provided an 
email address, through alumni Facebook pages, as an insert in our alumni magazine, and in the 
materials packet for all annual reunion attendees. All electronic requests for information 
included a link to the survey formatted in SurveyMonkey; the paper requests included a copy of 
the survey and a return envelope. The data collection began May 1, 2012 and continued until 
July 31, 2012. Respondents were asked to indicate their level of agreement with 25 statements 
designed to assess their opinion of the value and benefits of their Regis College degree such as 
“Regis prepared me to succeed in my career” and “the education I received at Regis was worth 
the investment in time and money” or to assess services such as “Regis provides valuable 
academic advising” and “Regis provides strong career development services.” They were also 
asked to rate 20 items indicating how much emphasis they believe Regis College should place on 
a variety of experiences or services such as career advising, study abroad opportunities, a broad 
liberal arts education, a global education, and faculty/student interaction outside of the 
classroom. Finally, respondents were asked to provide information about perceived importance 
of contributing to Regis, their contact information; year of graduation and major; information 
about additional degrees attained; employment status, employer and salary; volunteer and 
community service work; and accomplishments they are particularly proud of. The survey 


questions are provided in the Appendix. 
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Results 

Seven hundred ninety eight alumni returned surveys by the end of July, 2012. The vast 
majority of responses were electronic: 689 SurveyMonkey responses and 109 paper responses, 
with 76 of the paper surveys originating in our alumni magazine, and 33 in annual reunion 
packages. As shown in Figure 1, responses were well distributed across graduation decades, and 
the preference to respond electronically was equally distributed. A comparison of paper and 
electronic responses yielded no significant differences. Paper responders were not older than 
electronic responders, nor were they likely to show attitudes different from the electronic 
responders. 



The majority (59%) of survey respondents has either obtained an advanced degree, or is 
currently enrolled in a graduate program, as shown in Figure 2. 
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In addition to the 59% of respondents who already have at least one advanced degree, 30% of all 
respondents are considering furthering their education. The desire for additional education was 
most noticible among recent graduates, those who completed their education between 2007 and 
201 1, as shown in Figure 3. 


Figure 3. % of 2007 - 2011 graduates who are considering advanced education 



Figures 4 and 5 show salaries reported by the respondents. 
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Figure 4. Current salary of all survey respondents. 
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Figure 5. Current salary of 2007-2011 graduates. 
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Very, very few respondents reported that they were currently seeking employment, and most are 
employed full time, as shown in Table 1. 
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Table 1. Employment status 


All 

2007-2011 


Respondents 

Graduates 

Employed Full Time 

60% 

77% 

Employed Part Time 

13% 

17% 

Not Employed, Not Seeking Employment 

23% 

4% 

Not Employed, Seeking Employment 

4% 

2% 


Respondents provided an impressive list of professional accomplishments, 
service/volunteer/advocacy work, and recognitions/rewards/honors that they have received. A 
very brief sample, selected to show the diversity of experiences in provided in Table 2. 

Table 2. Sample accomplishments, advocacy work, and recognitions 

Professional Accomplishments 

Bringing several products to market in oncology and hematology 
I am known throughout the country as a tenured, proficient, Latin teacher 
Working with to help them combat childhood obesity 
COO of a Rehabilitation Hospital 
Being a member of the federal judiciary for 22 years 
Being executive director or a domestic violence/sexual assault program 
ServiceA^olunteer/Advocacy Work 

I am a guardian ad litem, representing the interests of abused/neglected children 
in court 

Dog Bones - Therapy Dog team 
Board of Directors; Historical Society 

Trustee of Family Foundation that supports grassroots community projects 
Working for the UN in peacekeeping operations 
Immigration Advocacy and Skill Training 

Recognitions, Awards, Honors 

National Park Service Star Award for helping to develop/pilot new education 
programs 

Resident of the year. Medical School Class President 
Army's White Plume Award for service to Soldiers and their Families 
Grant recipient from the Geraldine Dodge Foundation for work in education 
Elected first woman Mayor of my city 

Respondents were asked to indicate how important it is for them to contribute financially to 


Regis College and, if they do feel it is important, they were asked to describe their motivation for 
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giving. Table 3 shows attitudes toward giving for all respondents, and that 2007-201 1 graduates 
are more likely to feel that it is important. 


Table 3. Responses to, “How important is it for you to contribute financially 
to Regis College? 


All 

2007-2011 


Respondents 

Graduates 

Very Important 

29% 

38% 

Somewhat Important 

52% 

44% 

Not Important 

19% 

18% 


As shown in Table 4, reasons for giving varied a bit between all respondents and the most recent 
graduates. Recent graduates were less inclined to cite the desire to “give back” as their 
motivation, and slightly more inclined to cite peer encouragement, ensuring an excellent 
education for future students, and the tax benefits of charitable giving. 


Table 4. Motivation for Giving 


All 

Respondents 

2007-2011 

Graduates 

Regis College gave me an exceptional education; I want to 

give back. 

57% 

31% 

Fellow Regis College alums encourage me to give 

9% 

14% 

Regis needs my support to ensure an excellent education for 

future students 

60% 

63% 

The tax benefits of charitable giving 

13% 

20% 


In addition to giving financially, respondents indicated that they felt it was quite important to 
contribute to Regis College in other ways, with the younger alums most likely to feel this is 


important, as shown in Table 5. 
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Table 5. Responses to, “How important is it for you to contribute to Regis 
College in other ways? 


All 

2007-2011 


Respondents 

Graduates 

Very Important 

21% 

38% 

Somewhat Important 

53% 

44% 

Not Important 

26% 

18% 


Respondents were asked to indicate their agreement with a series of statements about Regis on a 
5 point likert scale. These responses provided the information we need to assess and improve 
our institutional effectiveness. Finally, they provided opinions regarding the things they believe 
we should be emphasizing going forward. 

Discussion 

The Regis College Alumni Survey provided invaluable content and methodological 
information for our institution, and for others considering implementing a similar survey. 
Responses were well distributed across graduation decades. The vast majority of responses 
(86%) were electronic. Interestingly, alums graduating in the 1940s were just as likely to 
respond electronically as our most recent graduates. Paper responders were not older than the 
electronic responders, nor did they reveal different attitudes than the electronic responders. This 
suggests that the use of on-line communication technologies has become sufficiently ubiquitous 
that we can use survey tools such as SurveyMonkey and assume that we are accessing a 
representative sample of our alumni population. We cannot, however, assume that a sample such 
as this is unbiased. All respondents were contacted through Regis Alumni sources. It is likely 
that we were contacting graduates who are motivated to stay in touch with Regis and our 
respondents were a self-selected group who were motivated to provide feedback. It is highly 
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likely that their opinions were more positive than a true random sample of graduates would hold. 
Given most of the goals of this survey — to improve our database of alumni information, identify 
internship and career opportunities for current students, obtain “success stories” for recruitment 
purposes, and obtain information that would support future fundraising efforts, this sample is 
appropriate. It is important to recognize, however, that this sample does not provide a true 
measure of institutional effectiveness. Never-the-less results did identify important institutional 
issues we should address and it did identify specific issues we need to obtain additional 
information about, using a more systematic sample. 

The majority of our respondents (59%) have obtained an advanced degree since 
graduating from Regis and most (69%) of our 2007-201 1 graduates are considering additional 
education, even though 39% of this group already has an advanced degree or are currently 
enrolled in a graduate program. We will be using this information to learn more about the 
graduate programs they are interested in, as we expand our graduate programming at Regis. 

Likewise, almost all of our respondents are employed. Only 4% overall and 2% of the 
2007-201 1 graduates reported that they are currently seeking employment. This is excellent 
news, but this is an area that is probably impacted by response bias. It is more likely that our 
graduates who are feeling successful would respond to the survey than those who are not feeling 
successful. The results must be interpreted with caution, therefore. Despite the probability that 
these results do not provide a full picture of alumni employment, they do provide an excellent 
source of information for Institutional Advancement, recruitment, and for career and networking 
opportunities or classroom speakers. Most respondents included specific information about their 
work and their employers. This is an invaluable resource for us going forward. Likewise, the 


salary information provided helps us as we discuss the paths our alums of various majors took 
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with our current and potential students. Information about professional accomplishments, 
service and advocacy work, and awards and recognitions is equally valuable. 

Not surprisingly, given our biased sample, the majority of respondents indicated that it is 
important for them to contribute financially to Regis College. It was surprising, however, to 
learn that our recent graduates were the ones who were the most likely to indicate that it is very 
important. Likewise, most respondents indicated that it was important to contribute in other 
ways and the young alums were again more likely to indicate that this is so. This is a significant 
finding. Young alums are telling us that they are ready, willing, and able to be involved. This is 
a relatively untapped resource. Although young alums usually have less financial means to give, 
it is important to establish a relationship and begin the mutually beneficial habits that will sustain 
both Regis and our graduates. Graduates who cannot give a great deal financially still have a 
much to offer experientially. Internship placements and career opportunities are very important 
to us. Further, we know that people who give in one manner tend to be more inclined to give in 
other ways. This means that alums who can provide opportunities or service now will be also be 
inclined to give financially when they can. Likewise, our financial donors are likely to provide 
additional types of experiential support if we ask. 

This survey has had an immediate impact on our operations at Regis. Based on the 
feedback that was much less positive than we would have liked we have already made dramatic 
changes to our career services and advising services. We have hired a new Director of Career 
and Internship Services and a task force is currently preparing a report on a revised academic 
advising model with a goal of implementing the new model in the fall of 2013. The alumni 
employment and experience information is being added to the career and internship database. 


Faculty have access to contact and experience information for classroom speakers, focus groups 
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and advisory boards. Institutional Advancement is taking full advantage of the updated contact 
information and the information about the alums who feel it is important to contribute to Regis. 
Information about graduate programs our alums are in and are considering attending is informing 
our own curriculum development as we consider how to best prepare our students for their future 
endeavors. Finally, we are using the information about what our graduates are doing in our 
admissions and recruitment messaging. 

We will be conducting follow up research to address some of the questions these results 
raised. For example, we want to learn more about the areas alums advised us to emphasize and 
about the areas in which they indicated that we have room for improvement. We will also be 
asking our respondents what we can or should do to foster relationships with our graduates, 
which we know are mutually beneficial. 

Although we do not plan to conduct another full scale survey such as this one for at least 
five years, we will be conducting targeted annual surveys. We will be getting in touch with 
graduates of specific majors as those programs undergo review, we will be following up with 
specific populations to learn more about how their experience compares with other students. 
Specifically, we want to learn about the experiences and opinions of graduates who are 
traditionally underserved, first generation, and/or low income. We want to learn all we can to 
ensure that we are maximizing the experience, and the probability of success, for future students 
from these vulnerable populations. 

We will be following up with specific academic ability groupings - do students who 
come to us with a high ability profile experience Regis and their life after graduation differently. 


or do we succeed at closing the gaps between these groups? Finally, we will use these results as 
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baseline data and follow up with specific questions so we can track institutional effectiveness 
trends. 


Future data collection will take advantage of electronic surveys. We will be holding 
focus groups, and we will intentionally contact a balanced sample of graduates in order to more 
accurately assess our ability to fulfill our mission. In addition, we will continue to work with 
those alumni who are interested and motivated to maintain contact with their alma mater. We 
have learned that alums, especially young alums, are an underutilized resource. They are 
motivated to give, and are telling us that they want to maintain a connection. 
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APPENDIX 

Regis College 2012 Alumni Survey 


Dear Regis Graduate, 

Your opinion is one of the most important indicators of the effectiveness of a Regis education. 

The perspective of alumni, 'who can evaluate hov^^ their experience has influenced their present 'work, 
activities, and beliefs, provides important feedback that 'we use to shape the things v^^e do for current 
and future students. Your perceptions help us gain a complete picture of hov^^ v^fell v^^e are doing and 
'what areas need improvements. 

Please take a few^ minutes to complete the accompanying Alumni Survey. Your responses and those 
of your former classmates 'will be used to advance the mission of Regis College and to guide and 
enrich the experience of future students. 

Please knov^^ that every response is significant and that v^^e value your perspective. 

Thank you for your support of your alma mater. 
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1. Please indicate your level of agreement with the following statements: 


Strongly 

Disagree 

Disagree 

Neither 

Agree 

or 

Disagree 

Agree 

Strongly 

Agree 

Regis prepared me to succeed in my career. 






If given the opportunity to start over, I would 
again choose to attend Regis. 






The education I received at Regis made a 
significant contrihution to the general quality 
of my life. 






The education I received at Regis deepened 
my understanding of moral and ethical 
problems. 






Regis empowered me to challenge myself 
academically. 






Regis empowered me to serve and to lead. 






Regis is a diverse and welcoming community. 






The education I received at Regis was worth 
the investment in time and money. 






Regis increased my appreciation of art, music, 
and theater. 






My spiritual life was enhanced at Regis. 






Since graduation I have felt an increasing 
loyalty to Regis. 






I would recommend Regis to others. 






Regis offers numerous opportunities to learn 
outside the classroom. 






Regis provides valuable academic advising. 






Regis provides strong career development 
services. 






Eaculty members at Regis are excellent. 






Regis provides strong support and attention to 
individual students. 






Regis should be concerned about the level of 
drinking and/or drug use on campus. 






Regis should be concerned about 
discrimination or harassment on campus. 






Regis should be concerned about declining 
academic standards. 
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2. Using a scale in which 1 is poor and 5 is excellent, how would you rate the following? 


Poor 




Excellent 

My overall experience with Regis College as a student. 

1 

2 

3 

4 

5 

The value of a Regis College degree in the job market. 

1 

2 

3 

4 

5 

My treatment as an alum. 

1 

2 

3 

4 

5 

My current feeling of connection to Regis College. 

1 

2 

3 

4 

5 


3. Using a scale in which 1 is “none at all” and 4 is “a great deal” please indicate how much 
emphasis you believe Regis College should place on each of the following: 


None 
At All 



A Great 
Deal 

Eaculty/student interaction outside the classroom 

1 

2 

3 

4 

Academic advising 

1 

2 

3 

4 

Career advising 

1 

2 

3 

4 

Leadership development 

1 

2 

3 

4 

Community service 

1 

2 

3 

4 

Internships 

1 

2 

3 

4 

A global education 

1 

2 

3 

4 

Study abroad opportunities 

1 

2 

3 

4 

Eaculty research 

1 

2 

3 

4 

Student research 

1 

2 

3 

4 

A broad liberal arts education 

1 

2 

3 

4 

Intercollegiate athletics 

1 

2 

3 

4 

Extracurricular activities (other than athletics) 

1 

2 

3 

4 

Commitment to Catholic mission 

1 

2 

3 

4 

Heritage of Sisters of Saint Joseph 

1 

2 

3 

4 

Campus Ministry activities 

1 

2 

3 

4 

Skills valuable in the workforce 

1 

2 

3 

4 

Moral/ethical development 

1 

2 

3 

4 

Need-based financial aid 

1 

2 

3 

4 

Merit-based financial aid 

1 

2 

3 

4 


4. In what year did you graduate? 

5. What was your major? 

6. What was your favorite class at Regis? 
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7. Please indicate the highest level of education you completed 

Bachelor’s degree 

Master’s degree 

Program of study, degree, and school 

Professional degree (Law, Medicine, etc) 

Program of study, degree, and school 

Doctorate 

Program of study, degree, and school 

8. Are you currently enrolled in a graduate program? 

Yes 

Program of study, degree, and school 

No 

9. Are you considering continuing your education? 

Yes 

Program and degree 

No 

10. Current employment status 

Employed full time 

Employer and position 

Employed part time 

Employer and position 

Not employed, seeking employment 

Not employed, not seeking employment 

11. Current Salary range 

I am not currently working 

Eess than $25,000 

$25,000 - $49,999 

$50,000 - $74,999 

$75,000 - $99,999 

$100,000 - $124,999 

$125,000 - $149,999 

$150,000 or over 

I prefer not to say 

12. Please share a professional accomplishment you are proud of. 

13. Please tell us about any community service/volunteer work/advocacy work you are involved 
with. 

14. Please tell us about any professional and/or community recognitions (awards or honors) since 
graduation. 
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15. How important is it for you to contribute financially to Regis College? 

Very important 

Somewhat important 

Not important 


16. If it is somewhat or very important to contribute financially to Regis College, which of the 
following describes your motivation to do so (indicate all that apply)? 

Regis College gave me an exceptional education; I want to give back. 

Fellow Regis College alums encourage me to give 

Regis College needs my support to ensure an excellent education for future 

students 

The tax benefits of charitable giving 

Other 

17. How important is it for you to contribute to Regis College in other ways? 

Very important 

Somewhat important 

Not important 

18. Please provide your preferred mailing address: 

19. Please provide your preferred email address 

20. Please provide your preferred telephone number 
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The NCES report of projections for 2020, include forecasts for several indicators and 
strata within the post-secondary educational system. A Holt-Winter filter model is pre- 
sented as a generalization of the NCES techniques with comparable results. Applications 
to enrollment and completions time series are discussed in the context of institutional 
research. 


1 Introduction 

NCES Reports focusing on projections, which include the Projections of Education 
Statistics to 2020 (Hussar and Bailey, 2011), use global and stratified time series to 
predict enrollments and completions, which are of particular importance for institutional 
research (IR). NCES projection techniques rely on classic concepts in time series analysis: 
linear models and smoothing. However, standard practices in educational research are 
focused on traditional cross-sectional linear models (Henson, Hull and Williams, 2010). 
This has a two-fold impact on the use of projections: projection figures may be mis- 
interpreted or underutilized, as the origins of such projections are obscure, due to the 
lack of familiarity with the techniques. Second, IR departments may be limited in the 
use of these techniques to make their own projections or adapt the national projection 
to specific contexts as needed. Here, NCES projection methodology is explained and 


1 



examples of its use are provided, as well as a comparison between technique variations 
and a discussion of the uses of these procedures in IR. NCES method (Hussar and Bailey 
,2011, p.78) includes two versions of an exponential smoothing procedure, and a linear 
trend model with external predictors. These techniques have complex extensions in the 
field of time series modeling with applications to institutional data (Weiler,1980;Li and 
Cheng, 2007; Cryer and Chan, 2008; Huang et. al, 2011).^ In this paper, the discussion is 
limited to the exponential smoothing techniques and a well know model that generalizes 
both specifications. NCES projections use other models, including linear equations for 
completions, and some of those equations include external regressors tied to population 
changes. The discussion here is limited to the smoothing techniques, given the wider 
range of applications to IR, in situations in which external regressors may not be avail- 
able or convenient to use. On the other hand, IR researchers may be more familiar with 
the regression-based models and may benefit more from an illustration of the smoothing 
techniques. 

1.1 Exponential Smoothing 

Exponential smoothing has two versions: single and double. These terms correspond to 
a more general formulation, that of Holt-Winters smoothing models (Winters, 1960). 

The single exponential smoothing procedure employed by the NCES is described as 
follows: 

at = axt + (1 - a)a(t-i) ( 1 ) 

with 

0 < a < 1 

, where at is the smoothed value at time t, Xt is the input value at time t, and a is 
the smoothing parameter. This model has been used extensively in several disciplines, 
and it is also known as the Exponentially Weighted Moving Average (EWMA) model. 
One important aspect of the EWMA model is its relationship with more general models 
for time series. In particular, EWMA can be considered equivalent to an ARIMA(0,1,1) 
specification with no Auto- Regressive (AR) component and a Moving Average (MA) co- 
efficient 1 — a, with l/a indicating the number of time points used for the MA (size of the 
moving window). In practice, some of the software implementations of the Holt-Winters 
model optimize the a parameter by minimizing the consecutive lead-out errors (from t 

^Time series analysis of educational data, specially enrollment series often involve highly tech- 
nical treatments. Fundamental issues such as stationarity, seasonality and transformations are not 
treated here due to space considerations and to keep the technical details at a minimum. The reader 
is encourage to review those topics in the relevant literature. 
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to t + 1), which can render a different parameter space than the standard ARIMA(0,1,1) 
using different constraints. There are practical ways to estimate the a parameter which 
are especially useful for spreadsheet implementations, and there are also rules of thumb 
on how to determine the a value (Cryer and Chan, 2008). One important property to 

keep in mind is that when a approaches 0, EWMA resembles a pure MA model, and 

when a approaches 1 becomes a random walk model. For most practical purposes the 
EWMA or exponential smoothing will deliver a different performance than these simpler 
models when the a parameter is far from the extremes of the parametric space. 

EWMA incorporates what is called a level component of the series, with the emphasis 
on the MA term. In the case of series with a strong trend component, the Holt-Winters 
generalization of the EWMA model provides away to incorporate the trend component 
into the model. An integrated level and trend smoother can be formed by adding the 
trend (slope) term to the EWMA, : 

o-t = (^{xt — St-p) -|- (1 — a){at-i bt-i) ( 2 ) 

bt = /3{at — flt-i) -f (1 — j3)bt-i (3) 

where bt is the slope at time t and j3 is the smoothing parameter. This is a Holt- 
Winters level-trend models with parameters [o;,/9], referred in the NCES reports as double 
exponential smoothing. In this case, the optimization of parameter a,/3 \s a more 
complex affair than for EWMA, requiring the identification of level and trend components 
of the series. Nevertheless, Holt-Winters models are implemented in most statistical 
software packages and may be used as smoothers even with spreadsheet software. 

In this paper, the a specification (level) is compared with the a, [3 specification (level- 
trend) for illustration purposes, in order to show how these different Holt-Winters specifi- 
cations can perform with some example data. These specifications are called exponential 
smoothing and double exponential smoothing in the NCES reports. 

1.2 Forecast Performance metrics 

The accuracy of the predictions are analyzed using the Mean Absolute Percentage Error 
(MAPE) metric, typically predicting the value of the series at tn-|-l, from the prior series 
values (to — t„). MAP error can be defined as: 


e, = 100 ^*1 (4) 

Xt 

where e* is the MAP error at time t, Xt is the observed value and Xt is the forecast value 
for time t. 
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Several aspects of this projection approach may be of special interest for the IR 
community: the interpretation and use the projection estimates across years, and the 
considerations to make when applying these projection techniques to their own data. 

This article illustrates the issues involved in the interpretation of the projections and 
presents a comparison of two projection techniques with national public data, later high- 
lighting how these techniques may be used with institution-specific data, making emphasis 
on the MAP error metric as used by the NCES projection report. There are many other 
forecasting error metrics, based upon model goodness of fit, prediction error, confidence 
band width, however the MAP error metric is used here to follow the NCES standards. 


2 Method 

2.1 Design 

This is a secondary data analysis study. Data is extracted from the NCES data sources, 
and the projection models are implemented to compare the projection results of the dif- 
ferent techniques. Variables Two time series were selected for the analysis: Enrollments 
in Public 4-year post-secondary institutions, and attainment in 4-year degrees in public 
post-secondary institutions. Each time series is accompanied by two stratification modi- 
fiers: gender and course load, both treated as categorical variables, to show the variability 
of the model performance across strata. The period of observation was 1991-2009, and 
the time-resolution was one data point per year. 

2.2 Analysis 

2.2.1 Comparison of projection techniques 

Projections of enrollment and degrees conferred were compared using the MAPE metric, 
for the whole series (1991-2009), obtaining the MAPE estimate for lagl (next year) 
projection for the full series. The models compared were the ERWMA and the Holt- 
Winters level-trend models. Following NCES report conventions , these are results are 
interpreted by their magnitude rather than statistical significance. 

2.2.2 Stability of the enrollment projections for post-secondary institutions 

Enrollment projection Mean Absolute Percent Error (MAPE) was analyzed for the 2011 
projected value using 10 time lags, as reported by NCES. This analysis was combined 
with an Auto-correlation analysis of the enrollment series using a 10 year window. Results 
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taken from the NCES reports are used for secondary analysis, and independent analyses 
performed by the author are presented, using Holt-Winters models evaluated for 1991- 
2001 series to forecast 2002-2009 values for enrollment and completions series from 
IPEDS data lab. 

3 Data sources 

All secondary data sources: NCES projection and Reports tables. Integrated Postsec- 
ondary Education Data System (IPEDS) Data Center (Enrollment and Completions Sur- 
veys) 


4 Results 

4.1 Comparison between the Techniques 

Several series were analyzed with the single and double exponential smoothing models, 
according to Holt-Winters specifications. These independent analyses were carried out on 
enrollment and completion series for national numbers in the period of 1991-2009. The 
results shown here are typically for models forecasting 2009 from the series 1991-2008 
(1 year lead-out projections). 

The single exponential (EWMA) model results show MAPE of 2.24% for Enrollments 
in Public 4 year institutions. For part-time enrollments, MAPE was 2.06 for females and 
2.26 for males. For completions, bachelor degrees conferred by public institutions, MAPE 
estimates were 2.6 for females and 2.33 for male students. In all these EWMA models, 
the optimal parameter estimate for a was 0.99. 

The Holt-Winters level-trend model (double exponential smoothing), for enrollments, 
yielded a MAPE estimate of 1.06%. For part-time, MAPE was 1.32 (Female), and 
1.38(Male). For Bachelor degrees, MAPE estimate was 1.28 (Female), and 1.73 (Male). 
For this model, parameter estimates showed greater variation: the enrollment series with 
a > 0.77 and b > 0.9 for all strata. For the completions series, a > 0.9 and b < 0.1 in all 
the cases. Generally speaking, the completions series have a stronger trend component 
than the enrollment series for this particular dataset. 

The MAPE differences between models are related to their goodness of fit. The 
level-trend Holt-Winters model shows better fit than the ERWMA, as shown in Figure 
1. The better fit for the a^(3 double-exponential model may be due to the presence of 
a strong trend on this completions series. As can be seen in the figure 1, the smoothed 
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Figure 1: Comparison of single and double exponential smoothing: Bachelor Degrees 
(Female) 1991-2009 


series for the level-trend model tends to be closer to the observed values, especially at 
the end of the time period, in which the linear trend is more dominant. 
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4.2 Stability of projection accuracy over time 

The prediction accuracy decreases over time (a standard result for time-series projections). 
MAPE doubles in size from 1 year lag (1.7%) to 3 years lag (3.9%), and multiplies by 
a factor of 7-1- at the 10 year lag (13.2%) (Hussar and Bailey, 2011, p. 119). This is a 
pattern which can be observed in many forecasting situations, but it is clearly illustrated 
by the NCES results. 


5000 


6000 




Figure 2: Lag-correlation plot: Enrollment in 4-year Public institutions 1991-2009. 
The circles represent observed data points, the center dotted line represent the pro- 
jected values and the solid lines the upper and lower-bound of the 95% conhdence 
interval (Cl) for the projection. 

One possibility would be to forecast the series using an auto- regressive model instead 
of exponential smoothing. However, further analysis reveals that such model would have 
similar issues in terms of long-term forecasting. 

Lag-scatterplots (Figure 2), show how auto-correlation decreases sharply with years, 
from a strong auto-correlation in lag 1, to a weak relationship by lag 10 (ten years lead- 
out), consistent with the decrease in the projections accuracy exhibited by the MAP error 
metric . In this case, an AR model will have the same declining forecasting accuracy over 
time, given the weak serial correlation observed in the series just after a few lags (years). 
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So far, the analysis have focused on point estimates of the forecast values. However, 
a more comprehensive analysis of the forecasting model can be accomplished by including 
the confidence interval the forecast, to put the forecasting accuracy in a better context. 

Figure 3 shows Holt-Winters level-trend model forecasting enrollment in 4-year public 
institutions. This plot shows forecasts for 2002-2009 based upon the 1991-2001 series. 
The projected values describe a linear trend, which decreases in accuracy until the very last 
year in which the observed point re-aligns with this trend. This highlights the difficulties 
involved in the evaluation of forecasting models using a single data point. On the other 
hand, the plot shows how the confidence band for the projection expands over time, 
making the forecast band much narrower for short lead-out times. 



Figure 3: Holt-Winters Forecasting: Enrollment in 4-year Public institutions 1991- 
2009. The circles represent observed data points, the center dotted line represent the 
projected values and the solid lines the upper and lower-bound of the 95% conhdence 
interval (Cl) of the projection. 

Figure 4 show how the presence of a strong trend component can stabilize the predic- 
tion accuracy, given that the model takes that trend into account, such as the level-trend 
Holt-Winters model used. It also shows how stratification plays a crucial role in the 
identification and interpretation of these forecasting models. For this case, while the 
confidence band widens over time, the projected value remain at a similar distance from 
the observed values, describing a rather parallel trend. In contrast with the prior series 


shown in Figure 3, the strong linear trend keeps the accuracy of the forecast relatively 
stable. 



YEAR 


2005 


Figure 4: Holt-Winters Forecasting: Bachelor Degrees (Female) 1991-2009. The 
circles represent observed data points, the center dotted line represent the projected 
values and the solid lines the upper and lower-bound of the 95% Cl. 

The MAP error values as shown in Figure 5 indicate how the short-term forecast error 
is very small when compared to the long-term forecasting error. In this case, there is an 
initial negative trend, and then a period with no clear observable trend. Just after 2005 
(the middle of the forecast period), the observed data start to show a positive trend. This 
result in MAP error estimates that grow from 2.9% in 2005 to 13.7% in 2009. When 
compared with Figure 4, this shows how the differences in the series, both the series used 
to fit the model and the one projected, can interact with the model specification to affect 
the stability of the forecasting accuracy. 

For illustration purposes, the model based upon the 1991-2001 can be called a long- 
term model or long-memory models.^. A much short-term model can be built using the 
2002-2006 series to forecast the 2007-2009 values. Such short-memory model won’t 
include the effects of the negative trend observed at the beginning of the series. 

^Long-memory models are a special kind of time series models, however, here the term is used 
just to draw the difference between the short and long-term impact on the forecast, and not in the 
strict sense of the definition of a long-memory model 
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Observed 



Figure 5: Holt-Winters Forecasting: Part-Time Enrollment (Male) 1991-2009. . The 
circles represent observed data points, the center dotted line represent the projected 
values and the solid lines the upper and lower-bound of the 95% Cl. The numbers 
right above the points are the respective MAP error estimates for each projected 
value. 


The contrast between the long and short-memory models is illustrated in Figure 
6, which include projected series and confidence bands for both models. The shorter, 
narrower projection band covering just the 3 final point of the series corresponds to 
the short-memory model. This plot shows that even with the short memoy model, the 
changes in the series makes forecasting a difficult task for this kind of model. The 
projected series is much closer to the observed values but 2 of the 3 observed values fall 
outside the confidence band for the forecast. This may be an indication that a different 
kind of model is required for this series, and illustrates once again how stratification can 
affect modeling and forecasting. 


5 Discussion 

NCES projection reports illustrate two important properties of time series forecasts: sen- 
sitivity to stratification and differential accuracy of short-term vs long-term projections. 
Projection accuracy decreases as the lag between the observed and projected value in- 
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Figure 6: Comparison of long and short-memory Holt-Winters Forecasting with con- 
hdence band: Part-Time Enrollment (Male) 1991-2009 

creases. This is a regular occurrence in time series projections, even with the large sample 
size. On the other hand, given how the Holt-Winters model parameters are optimized, 
short-term projections are favored by such procedure. 

Caution should be exercised when making projections of enrollment and completions 
of particular institutions, or even interpreting the NCES projections, given that accuracy 
decreases sharply over time. Most series can be predicted within a 2% MAPE for the 
first year, but MAPE increases after 3-|- years. The Judgment about an acceptable error 
will depend on the institutional priorities and the sample size, which can have a strong 
influence the accuracy of the predictions. As to the different projection techniques, it 
is important to consider the combination of techniques and stratification. The same 
stratification variable can modify the accuracy of projections in different directions: for 
the ERWMA model, gender stratification affects the accuracy of predictions in opposite 
directions for enrollments vs completions. Overall, the Holt-Winters level-trend model, 
combining the MA and slope components, performs better with the NCES data. The use 
of NCES projections as reference values is recommended with the relevant stratification 
(institution type, sector, demographics). 

It is important to recognize that several of the examples presented here contain a 
strong trend component, thus favoring the two parameter version of the model. For 
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practical applications in IR, checking both versions of the model could be a good ap- 
proach, as with single-institution data with no strong trend, the EWMA model may 
provide a more parsimonious specification. 

The NCES projection techniques can be applied using institution specific numbers 
to create statistically sound forecasts. These techniques can be applied to enrollment, 
completions and graduation and retention rates treated as time series. 


6 Final Comments 

The exponential smoothing projection approach provides the institutional researcher with 
a flexible tool to make statistically sound projections, with applications to enrollments 
and completions. The NCES projections are a very useful reference for IR scientists and 
practitioners, showing the robustness of the methodology and applicability across multiple 
strata. 

Some necessary adaptations need to be made in order to apply these techniques to 
a specific institutions. There are important differences in the sheer volume of data, and 
the magnitude of the quantities handled. On the other hand, for a national series it is 
more meaningful to use external indicators (i.e. census data) in regression-type models 
for forecasting. For the specific institution, the relationship of those external regressors 
to the local numbers can be more indirect or delayed. 

Another consideration that goes beyond this paper is using different time resolutions 
such as monthly or quarterly data. In such cases there’s an additional component to be 
considered which is seasonality. In such cases, the Holt-Winters model can accommodate 
the seasonal component. 

Regardless of the model chosen, projections are statistical estimates, which should 
be interpreted in the context of a confidence band. It should be highlighted that the 
projection error is an integral part of the model-building process, as stated by the NCES 
report: 


Projections of a time series usually differ from the final reported data 
due to errors from many sources, such as the properties of the projection 
methodologies, which depend on the validity of many assumptions. (Hussar 
and Bailey 2011, p. 1.) 

Forecasting models are complex dynamic analytic components which need to be main- 
tained and revisited on a regular basis, and forecasting error is just one element of quanti- 
tative feedback that must be used to improve the model. Given the relative simplicity of 
the Holt-Winters model, and the ability to make projections without external predictors. 
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this segment of the NCES methodology has shown to be useful for IR, provided that is 
used for short-term forecasting, and with the necessary caveats about the difference in 
the volumes of data. 


13 



References 


1. Cryer, J. D. ; Chan, K.S. (2008). Time series analysis. Second Edition. New 
York: Springer. 

2. Henson, R. K.; Hull, D. M. ; Williams, C. S. (2010). Methodology in Our Edu- 
cation Research Culture : Toward a Stronger Collective Quantitative Proficiency. 

Educational Researcher, 39, 229-240 

3. Huang, Y.; Horng S.; He ,M; Fan, P.; Kao T.; Khan, M.; Lai, J.; Kuo, I. (2011) A 
hybrid forecasting model for enrollments based on aggregated fuzzy time series and 
particle swarm optimization. Expert Systems with Applications, 38, 8014-8023. 

4. Hussar, W.J., and Bailey, T.M. (2011). Projections of Education Statistics to 
2020 (NCES 2011-026). U.S. Department of Education, National Center for Edu- 
cation Statistics. Washington, DC: U.S. Government Printing Office. 

5. Li, S.-T.; Cheng, Y.-C. (2007). Deterministic fuzzy time series model for forecast- 
ing enrollments. Computers and Mathematics with Applications, 53, 1904-1920. 

6. Weiler, W. C. A Model for Short-Term Institutional Enrollment Forecasting. The 

Journal of Higher Education, 51,314-327. 

7. Winters, P. (1960). Forecasting sales by exponentially weighted moving averages. 

Management Science, 6:324-342. 


14 



1 


IR Report 

Non-returner Survey: Why Students Leave 

Laura Ariovich and W. Allen Richman 

Prince George’s Community College 

310 Largo Rd 
Largo, MD 20774 


Abstract 

Prince George’s Community College has regularly conducted the Non-Returner Survey since 
2001. The survey is administered in the Fall to students who were enrolled in the previous 
Spring, but did not enroll or graduate in the following Fall semester. A new methodology used in 
Fall 2011 raised the survey response rate, saved resources, and yielded, for the first time, an 
actionable data set. 

Introduction 

Prince George’s Community College (PGCC) is a large suburban college located in Largo, 
Maryland, with a predominantly African American population. Offering a broad range of credit 
and non-credit programs, the College attracts over 40,000 credit and noncredit students annually. 
In Fall 2011, the College had a headcount of 14,647 credit students. The majority of these 
students were part-time (70%) and female (60%). 

As do other Maryland community colleges. Prince George’s Community College periodically 
surveys its non-returners, defined as those credit students who were enrolled in the Spring but 
did not return in the following Fall. There are two questions in the survey that are required for 
State reporting: 

1. What was your main goal in attending PGCC in Spring 2011? 

2. Do you believe you achieved the above goal at PGCC? 


In the past, the methodology used by the Office of Planning, Assessment, and Institutional 
Research (OPAIR) consisted of identifying all non-retumers and then drawing a random sample 
of 1,000 students, who would receive the survey questionnaire by postal mail. Students who did 
not reply would receive a second mailing. Even with the second mailing, the survey had very low 
response rates. Between 84 and 160 respondents — or about 1.7-3% of the actual non-returner 
population — filled out the survey between 2001 and 2007. The low response rate severely 
limited the survey’s utility and did not help identify non-retumers’ reasons for leaving. 
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Moreover, the survey had little use to inform interventions aimed at reducing the non-retumer 
population. 

The purpose of this report is to underscore the importance of the non-retumer survey in the 
context of the typical enrollment flow at PGCC, present a revamped methodology to survey non- 
returners, and discuss the survey findings and next steps to reduce the non-retumer rate. 

PGCC Enrollment “Flow” and the Importance of Studying Non-Returners 
As background research for the survey, OPAIR examined the College’s enrollment trends from 
2007 to 2012. As seen in Figure 1, PGCC’s Fall enrollment has been fairly consistent over time, 
notwithstanding a slight annual increase from 2008 to 2010. 


Figure 1: PGCC Fall Credit Enrollment 



However, the stability of overall enrollment trends is deceptive when the “flow” of students 
enrolling and not returning is not examined. An examination of the typical enrollment flow from 
the Spring to the Fall semester for 2007-2011 showed that each Spring the College loses 5,000 to 
6,000 students, approaching 40% of the credit population. Despite the massive Spring exodus, 
the College manages to maintain its overall enrollment in the Fall by “replacing” the lost 
students with three sources of incoming students: students new to College, students who transfer 
into PGCC from other institutions, and students who renew their attendance after having stopped 
for one or more semesters (see Figure 2). 
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Figure 2: PGCC Enrollment Flow 


Typical Emollment Flow at PGCC 

(estimates based on 2007-2011 Fall data) 


Spring Semester 
Enrolled Students 


New to 


Transfer-ins 



Fall Semester 
Enrolled Students 


Stop outs. 
Drop outs, & 
Transfer outs 


Analyzing the student enrollment flow brought to the fore the previously overlooked loss and 
“replacement” of students from Spring to Fall. These data highlighted the need to learn more 
about the non-retumer student population and take steps to reduce the non-retumer rate. 


Revamping the Non-Returner Survey 

In fall 2011, OPAIR revamped the Non-returner Survey with the purpose of acquiring useful and 
actionable data. Instead of sending the survey through postal mail to a random sample of 1,000 
students, an electronic survey was sent to all non-returning students with valid email addresses. 
After the initial email, four reminders were sent to those who had not yet responded. Reminders 
were sent on different days of the week and at different times during the day. All reminders 
produced an initial jump in the number of responses (though the size of this jump diminished 
over time). 

In order to increase the Non-Returner Survey response rate, OPAIR relied on two strategies. The 
initial invitation to fill out the survey and all reminders were directly signed by PGCC President. 
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In addition, all respondents who filled out the survey within the first 25 days were entered into a 
raffle to win a $50 gift card (with the exception of those who were employed by PGCC at the 
time of the survey). The survey was open for a month, but the majority of respondents (close to 
72%) filled out the survey by the deadline for the raffle. 


Response Rate 

The Non-returning Student Survey was sent by email as an electronic questionnaire. Out of the 
initial population of students who did not return in Fall 2011, 6,265 actually received the 
questionnaire; the rest (169 students) had no email addresses on file or had invalid email 
addresses. From this pool, 964 (15.4%) responded, yielding more than six times the best 
response rate of previous non-retumer surveys. An analysis of demographic and other data for 
the 964 respondents showed that: 

• 70% were female 

• They had a mean age of 34 

• They attended a median of 6 semesters at PGCC 

• They completed a median of 26 hours at PGCC 

• 40% were currently attending another institution 

• 70% were working 

• 60% planned to return to PGCC 


Although only the first three questions in the questionnaire were required, 96% of respondents 
completed the full survey. In addition, 65% of respondents chose to answer the open-ended 
question: “In what ways could PGCC better serve your educational needs?” As shown by these 
comments and by direct email replies sent by respondents, students valued the survey as an 
opportunity to convey their experiences, offer suggestions, and even ask for assistance to resume 
their college attendance. 


Findings: What we learned about non-returners 

An analysis of the mandatory questions showed that nearly 50% of non-retumers came to PGCC 
to earn an associate’s degree, whereas smaller percentages came to transfer (26%) or for other 
reasons (26%). Among those who came to PGCC for an associate’s degree, one third (33%) did 
not achieve their goal, one third partially achieved their goal, and one third achieved their goal 
completely (see Figure 3). Among those who came to PGCC to transfer or for other reasons (e.g., 
update skills, personal enrichment, etc.), the rate of students not completing their goals was 
significantly smaller, around 15%. Thus, more non-returners came to PGCC to obtain an 



associate’s than to transfer or for other reasons. However, those seeking an associate’s were 
significantly less likely to achieve their academic goal. 

Figure 3 
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In addition to the mandatory questions, the survey asked students about any hurdles which may 
have prevented them from returning to the college in the Fall semester. The students ranked these 
potential barriers as “major reasons,” “minor reasons,” and “not a reason” for not enrolling in the 
Fall (see Table 1). Approximately one third of respondents reported leaving because they had 
achieved their academic goal or transferred to another institution. Examining only those students 
who reported leaving without obtaining their educational goal or transferring, the factors 
identified by most students as a “major reason” for leaving were not having money to enroll 
(43%), personal problems (38%), being unhappy with one’s academic progress (25%), and being 
unhappy with the College’s services for students (25%). 


Table 1 


Item 

% Major Reason 

I did not have the money to enroll. 

43 

I could not attend college due to personal problems. 

38 

I was unhappy with my academic progress. 

26 

I was unhappy with the services for students at PGCC. 

25 
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Equally important, the survey helped identify factors that were not perceived by students as 
reasons for leaving. Surprisingly, these factors included being unable to find childcare, lack of 
transportation, moving away from the area, and being unhappy with activities for students at the 
College. In all these cases, 85% of respondents or more indicated that the listed item was “not a 
reason” for leaving (see Table 2). 


Table 2 


Item 

% Not a Reason 

I could not find child care so that I could attend classes. 

88 

I could not get to campus due to lack of transportation. 

85 

I moved away from the area. 

85 

I needed a break from school. 

77 

The classes or programs I wanted were not available. 

76 

My educational goal changed. 

75 


Student Feedback in response to the Open-Ended Question 

As stated earlier, one of the impressive elements of the 2011 Non-Returner Survey dataset was 
that 65% of respondents typed answers to the open-ended question. A quick review of open- 
ended responses showed that most were detailed and somewhat lengthy. Given the richness of 
these data, students’ comments were coded and analyzed with NVivo, a qualitative research 
software program. The analysis was conducted in three stages: 

Stage 1 : Tentative categories were formulated based on a preliminary review of all 
comments. 

Stage 2 : All comments were coded based on the tentative categories. As part of this 
process, the categories were reformulated to achieve a better fit with the data. 

Stage 3 : Some categories were merged and others eliminated based on the type of 
comments and the number of quotes included in each category. All comments were then 
recoded using the final set of categories. The analysis focuses on the four categories in 
the final set that contained the larger number of comments. 

The results of this analysis are summarized in Table 3. As shown in the table, roughly a quarter 
of comments reflected positive experiences with the College. However, there were also areas 
where students found barriers to success. Slightly over 25% of comments cited lack of 
responsiveness to student issues and concerns, including problems in specific areas such as 
advising and financial aid as well as more generalized concerns with interdepartmental 
communication and customer service. A much smaller but still sizeable set of concerns (13%) 
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addressed the lack of available courses at times that would fit students’ schedules. The last set of 
comments (8%) had to do with students’ difficulties paying for college and requests for greater 
flexibility or assistance in this area. 


Table 3: Analysis of students’ comments (summary) 


Main categories 
(may overlaB) 

Percent of all comments 

Subcategories within main category 
(may overlaB) 

Positive comments about 
PGCC 

26% 

Goal achievement 

Encompassing positive experience 

Positive experiences with faculty 

No complaints 

Eack of college 
responsiveness 

28% 

Problems with academic advising 

Problems with financial aid 

Problems with faculty 

General problems with college 
responsiveness 

Eack of course availability 

13% 

Locations, times, days, and frequency 

More online or hybrid courses 

Affordability 

8% 



Students’ responses to the open-ended question offered a unique glance at their experiences with 
academic and nonacademic services. Understanding non-retumers’ perspectives is an essential 
step in any successful effort to remove hurdles to student success. A more detailed account of the 
main classes of comments and subcategories is presented below. 


Main category: Positive Comments 

Even though the wording of the question prompted respondents to think about ways to improve 
PGCC, over a quarter of those who did respond (26%) had positive things to say about the 
college. There were four subcategories of positive comments: 

• Subcategory 1: Goal achievement 

Some students thanked the college for helping them achieve their educational goals. 
Students in this group had a variety of goals: preparing for transfer to a four- year 
university, completing one course or a few courses for a degree pursued at another 
institution, completing a class needed for teacher certification, or simply updating their 
skills. 




Subcategory 2: Encompassing positive experience 
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Some students declared explicitly that they had a positive experience at PGCC. In some 
cases, students praised the institution as a whole without providing further details. In 
other cases, students detailed specific aspects that they valued at the college, including 
their classes, the service received, the broad range of programs, and the flexible schedule. 

• Subcategory 3: Positive experiences with faculty 

Some students emphasized their positive experiences with faculty. Students in this group 
thanked PGCC faculty for believing in them, for helping them achieve their educational 
and career goals, and for assisting them with specific course challenges and skills. 

• Subcategory 4: No complaints 

Some students simply declared or implied that they had no complaints about their 
experience at PGCC. Students in this group used words such as “fine,” “OK,” and “fair” 
to characterize PGCC and/or their experience at the college. In some cases, students 
clarified that it was external reasons or personal problems, rather than something to do 
with the institution itself, which led to a temporary break in college attendance. 


Main Category: Lack of College responsiveness 

Over a quarter of those who responded to the open-ended question (28%) were not satisfied with 
the level or the quality of support they received at the college. In some cases, students 
complained about the customer service received at the college. In others, students expressed a 
need for greater empathy, additional help, or stronger guidance as they negotiated the challenges 
of their academic and personal lives. There were four subcategories of comments about the lack 
of college responsiveness: 

• Subcategory 1 : Problems with academic advising 

Some students complained about their experience with academic advising. Students 
expressed dissatisfaction with what they perceived as unfriendly staff and long waits. 
They also complained about having received incorrect advice as well as inconsistent 
advice from different advisors. Finally, students regretted having received insufficient 
guidance, with negative consequences down the road. 

• Subcategory 2: Problems with financial aid 

As it happened with academic advising, some students were dissatisfied with what they 
perceived as unfriendly staff and long waiting times in the financial aid office. Others 
complained about insufficient guidance regarding the application process, eligibility 
criteria, and funding opportunities. In addition, students complained about not having 
received proper notification regarding important requirements and deadlines. Finally, 



9 


students perceived a lack of empathy and leniency towards those facing difficult life 
circumstances. 

• Subcategory 3: Problems with faculty 

Some students complained about their experiences with faculty. Students expressed 
dissatisfaction with what they perceived as a lack of connection between students and 
faculty. In their view, some faculty members knew the materials but did not know or did 
not care enough about their students. Students complained also about insufficient support 
and guidance with in-class learning and take-home assignments. Finally, some students 
who struggled with academic or personal problems expressed a need for greater empathy 
and understanding from instructors. 

• Subcategory 4: General problems with college responsiveness 

Some respondents complained about poor customer service and inadequate support for 
students in specific areas outside of academic advising, financial aid, or the classroom. 
Others felt that the problems with college responsiveness were systemic and thus 
exceeded a particular group of people or a specific department. 


Main Category: Lack of Course availability 

Almost 13% of those who responded to the open-ended question had suggestions and complaints 
about course availability. There were two subcategories of comments about course availability: 

• Subcategory 1: Locations, times, days, and frequency 

For some students, it was a problem that classes were not available at specific locations, 
at specific times (especially in the evening), or on specific days to better fit their 
schedule. Other students expressed frustration at the fact that courses required for their 
programs were not offered regularly, which unnecessarily extended the time needed for 
completion. Students who expressed concerns about course availability often mentioned 
these problems in reference to upper-level courses or courses needed in the later stages of 
their programs. 

• Subcategory 2: More online or hybrid courses 

Some students expressed an interest in greater availability of online or hybrid courses. It 
is important to note that within the whole group of comments about course availability, 
comments referring specifically to online or hybrid courses were in the minority. 
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Main Category: Affordability 

Close to 8% of those who responded to the open-ended question were concerned about college 
affordability and most of them expressed a need for greater financial assistance. In particular, 
students were interested in grants, scholarships, and work/study options. Some of these 
comments reflect a need for better information and guidance about available financial aid 
opportunities and the process of applying for grants and scholarships. Some students may not be 
aware of the competitive nature and stringent requirements often associated with the disbursal of 
grant funds. 


Summary and next steps 

Since 2001, PGCC has surveyed non-retumers, defined as those credit students who had attended 
classes in the Spring but did not re-enroll the following Fall. However, the College had 
approached the survey more as a mandatory reporting requirement than as an intentional effort to 
collect actionable data. 

In 201 1, OPAIR conducted an analysis of the typical student flow at PGCC as background 
research for that year’s Non-Returner Survey. This analysis showed that 5,000 to 6,000 students 
enrolled in the Spring typically leave the following Fall. Interventions which might retain even 
10-20% of the 6,000 potential non-retumers would result in a 4-8% increase in Fall enrollments. 
Furthermore, aiding students who are at risk of exiting across these two semesters would likely 
have an impact on completion rates as well. 

In order to learn more about non-retumers, OPAIR adopted a new methodology to conduct the 
2011 Non-Returner Survey. The new methodology dramatically increased the response rate from 
the low single digits to 15.4%. Furthermore, the survey produced high-quality data as shown by 
the high proportion of students completing the survey and providing detailed comments in 
response to the open-ended question. The new methodology had the additional advantage of 
saving resources when compared to the traditional method of sending repeated mailings to a 
subset of non-returners. 

By ensuring high student participation and engagement with the survey, the new methodology 
produced a wealth of information about non-returners’ experiences with academic and 
nonacademic services. In addition, the survey helped to identify major hurdles and constraints 
faced by these students, which could inform interventions to reduce the non-returner rate. The 
ensuing follow-up actions were proposed: 

1. Focus on removing barriers in those areas identified by most students as “major reasons” for 
not returning, including assisting students in applying and staying eligible for financial aid, 
helping students experiencing lack of academic progress, and improving nonacademic 


services. 
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2. Improve internal communication and customer service to make the college more 
“responsive” to student issues and concerns. 

3. Reach out to students before they leave. The Non-Returner Survey focuses on students who 
have already stopped attending the College, at least temporarily. As a complement, an 
“intervention survey” should be conducted in the Spring to identify those students who are 
considering not returning, connect them with College services, and remove barriers to their 
continued enrollment and success. 
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Introduction 

Effective assessment of perceptions of stakeholders about college mission and identity 
provides insight into whether an institution's vision and purpose are being realized. Perceptions 
of staff, students, and faculty of their university can influence how fully each group engages in 
their community (Ferrari and Velcoff, 2004). This research addresses the need to go beyond the 
frequently-studied student population and measure faculty and staff groups on their 
perceptions of an institution's effectiveness in achieving overarching goals. This is especially 
important since it is those individuals who realize the seemingly esoteric goals of the institution 
in the practical contexts of their work on campus. 

The purpose of this research is to measure the perceptions of institutional mission of 
two groups, staff and faculty, at two universities and assess performance based on individual 
results and by a comparison of outcomes by institution and by demographic characteristics. The 
research is described in two phases. The first covers the research design and administration of 
the instrument, data collection, and preliminary analysis of results. That work was previously 
presented in a Workshare session at the North East Association of Institutional Research 
(NEAIR) Conference 2011. The second phase consists of additional analysis of data and a 
discussion of results and implications. 
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The research is in part framed by the theory of the two-step flow of communication 


(Katz & Lazarsfeld, 1955). It examines whether the mission and identity of the institution is 
being communicated adequately by leadership to constituents by virtue of the goals and 
programs the institution pursues and whether that is discernible in data obtained on relevant 
perceptions of staff and faculty. Results reported here help to illuminate how opinion leaders 
influence reception of the message of mission and identity by audiences of staff of faculty and 
what existing or proposed programming is perceived as supporting mission and identity. 


Two-Step Flow Model 



O = Opinion leader 

\ 

O = Individuals in social 
contact with an 
opinion leader. 

Figure 1: Reimagined Two-Step Flow Model for the purposes of the current research 

This research is also informed by Engagement Theory (Kuh, G.D., Pace, C.R. & Vesper, N, 
1997). Findings from research on student engagement suggest that what an institution is, its 
profile, matters less than what it does, for example, in instruction, programming and campus 
interactions. Engagement theory holds true for faculty and staff, as well, in that their 
perceptions of mission and identity are influenced not just by static words or published 
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assertions, but by the active commitment and initiative they observe in the surrounding 
university environment. 

Mission statements of institutions often include similar purposes and constructs, but the 
interpretation and practical applications by campus are unique and affect groups differently. 
Therefore, it can be expected that not all campus constituencies will share the same 
perceptions of purpose and identity, so this research intends to elucidate whether and to what 
extent differences by group do exist. There has been previous research using the DMV 
Inventory with a student sample at a single institution (Ferrari et al., 2009), but this is the first 
study to conduct a quantitative assessment of faculty and staff perceptions of mission at two 
different religiously-affiliated universities in eastern United States. 

This quantitative research involved data collection on perceptions of mission and 
identity of staff and faculty at two private, religiously-affiliated postsecondary institutions and 
subsequent analysis of results by institution and between institutions. Data was obtained by 
administering the DePaul Mission and Values Inventory (DMV) to the two groups. Analysis of 
results lends perspective to understanding current perceptions within institutions and the 
extent and nature of differences between groups and institutions. The discussion sheds light on 
the impact of institutional initiatives around mission and identity on perceptions of constituents 
and aids our understanding of how community cohesiveness on campus relative to goals might 
be enriched. 

Background 

Content analysis of mission statements by Abelman & Dalessandro (2009) reveals similar 
themes across many institutions, as well as the consistent use of what the researchers call 
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"visionary" terminology. Despite similarities, a variety of approaches, both strategic and 
programmatic, are taken by institutions to carry out mission imperatives. In the same way, 
even though many characteristics and goal statements of identity and mission at the 
institutions in this study are similar, this research investigates whether differences exist in 
perceptions of their separate constituencies. Consider the finding of Borne et al. (2000) that 
institutions that clearly articulate their missions are more effective at strategic planning. If so, 
the appearance in this research of differences in perceptions between the institutions may 
perhaps be accounted for by programs and activities related to strategic plans, which are 
essentially derived from institutional mission statements. 

Research suggests that maintaining identity can be positive for an institution of higher 
education (Estanek, James & Norton, 2006). A longitudinal study of 824 private colleges from 
1975 to the present found that religious colleges are more likely to survive than secular 
institutions (Porter and Ramirez, 2009). Administrators are taking notice. For example, at the 
2009 conference of the Association of Catholic Colleges and Universities (ACCU), the focus of 
presidents gathered there was on two issues they believe most important to their institutions 
today: the United States economy and its impact on students and the Catholic identities of their 
institutions (Ziegler, 2009). Thus, administrators of Catholic institutions understand the 
importance of communicating and sustaining the mission and identities of the institutions they 
serve. 

Morphew and Hartley (2006) point out that mission, regardless of an institution's 
affiliation, helps distinguish institutional imperatives and has the capacity to inspire and 
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motivate those within to communicate its characteristics, values and history to key external 


constituents. "Ultimately, the life force of any institution is its mission" (Scott, 2006). 

Estanek, James, and Norton (2006) said the optimum way to assess mission and identity 
in higher education is "best grounded in the mission of the individual institutions" (p. 205). 
Mission statements from a host of colleges across the nation were examined using content 
analysis. It was concluded that, since mission statements firmly convey the goals and objectives 
of a postsecondary institution, they could appropriately be employed as bases for assessing the 
presence of mission and identity within the community and programs of institutions. Prior 
research recommends that institutions communicate mission via administrative operations, 
student services, academic programs and student services (Ferrari and Cowman, 2004). 

One instrument being used to study mission and identity is the Mission and Values 
Inventory, which was developed by researchers at DePaul University in Chicago. Initially, Ferrari 
and Velcoff (2006) surveyed more than five hundred staff members there and found the 
majority of those surveyed understood DePaul's mission. Catholic values, and mission-driven 
activities. Results also indicated that the majority of staff members surveyed believed that 
institutional mission is conveyed throughout the institution via the administration, academics, 
policies, and student services (Ferrari and Velcoff, 2006). 

Assumptions and Limitations 

It is assumed that the DMV Inventory is an effective tool for obtaining reliable data on 
faculty and staff perceptions of mission and identity at postsecondary institutions. It is further 
assumed that employing a subset of mission and identity-focused questions from the DMV and 
not the entire inventory will nonetheless generate sound results adequate to the purposes of 
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this research. It should be noted that a limitation of the DMV is that some items pose two 
questions in one statement. Despite many similarities in the institutional characteristics and 
mission statements of the colleges involved in the study, it is assumed that some differences in 
group perceptions do exist and are discernible using the methodology proposed. 

This research is limited to a sample of two postsecondary institutions. Each institution is 
religiously-affiliated, although with different orders of religious. Survey participants were 
limited to faculty and staff at the institutions, so no comparative student data was obtained. 
Additionally, the medium of communication to prospective respondents was limited at one 
university to a single email message distribution to faculty and staff and did not allow for 
distribution of any follow-up or reminder message. The survey administration was electronic, 
only, limiting the survey distribution to only those with ready access to a computer. 

Importance 

This research contributes to scholarship in the areas of institutional mission 
effectiveness, strategic planning, program assessment and engagement. Assessing perceptions 
of mission and identity by these groups can give evidence to affirm or discount the assertions 
by Borne et.al (2000) that institutions clearly articulating their missions are more effective at 
strategic planning and at planning for growth and enhancement of the institution (Amis et.al, 
2000, Detomasi, 1995). 

The study yields data on staff and faculty groups from institutions in the eastern region 
of the nation. Previously, only institutions in the Midwest had been investigated using portions 
of this instrument. Additionally, results of this research are able to inform accreditation and 
strategic planning reporting needs of the institutions in the study in new ways by providing 
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institution-level data of a kind not previously available on mission effectiveness in 
postsecondary education. The procedure for this study can be imitated by other institutions for 
accreditation purposes, as well. Lastly, findings of this study provide insight on the potential 
local and regional impact of university staff and faculty who carrying forward in their lives at 
work and in the greater community the positive behaviors and attitudes embodied in the 
missions of their institutions. 


Methodology 

The methodology for this research is quantitative and the instrument used to gather 
data is the DePaul Mission and Values Inventory (DMV) developed by Ferrari and Velcoff, 
(2006). Sixteen question items (Appendix A) were drawn from the 39-item DMV Inventory and 
administered to the sample. Demographic items were added by the researchers (Appendix B). 

The instrument was administered online in spring 2011 to faculty and staff groups using 
the Survey Monkey online survey platform. The invitation to participate was delivered to 
employees at both campuses via campus email that contained the link to the online Survey 
Monkey site. At the conclusion of the administration, data were obtained from the password- 


protected Survey Monkey account of the researchers and loaded into SPSS for later analysis. 

Procedure 

O Select two "like" institutions for the study 

• populations 

• mission statements 

O Target employee samples (by position) 

• Faculty 

• Staff 

O Customize the DePaul Mission and Values 
Inventory (DMV) 

• Use a Likert response mode 1-7 

• Select only mission and identity items (16) 
for use from all DMV items (39) 

O Distribute invitation via employee email 
O Administer online using Survey Monkey 

https://www.survevmonkey.eom/s/DGNJ3BF 

O Analyze results using SPSS 
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All part-time and full-time members of the staff and faculty (n=2,021) at both 


institutions were invited to participate in the study via an email invitation containing a web link 
to the survey, available on the Survey Monkey platform. The survey requested that 
respondents rank the 16 mission and identity-related items according to a 7-point Likert scale 
ranging from strongly agree to strongly disagree. Mean scores were produced by item. Artificial 
constructs were created on an a priori basis to allow comparison of construct means by 
institutions and groups using more robust factor scores. 

Participants 

The survey population consisted of all full-time and part-time employees of University A 
(n = 870) and University B (n = 1,151). There were 364 respondents to the survey, signifying an 
18 percent response rate overall. The distribution of the 364 respondents by institution is 
University A, n = 206 (24 percent) and University B, n = 158 (14 percent). 

Data 

Prior to performing frequency analysis, data were reverse coded on responses according 
to the DMV Inventory scale of l=strongly agree, 7=strongly disagree to l=strongly disagree to 
7=strongly agree. This would parallel our cultural inclination to perceive higher scores as 
"strong" and lower ones, "weak." Therefore, results showing higher means by item would 
indicate better performance relative to lower means. Frequency analysis indicated that the 
distribution of respondents by position at both institutions was 54 percent staff and 44 percent 
faculty, with 2 percent unknown. Employment status of respondents is 86 percent full-time and 
14 percent part-time. By gender, 69 percent of respondents are female and 31 percent are 
male. Just 3 percent of respondents are non-Caucasian 
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Table 1: Demographic characteristics by institution 



A 

B 

Characte ristic 

|N = 206) 
% 

(N =158) 
% 

Position 



Facu Ity 

44 

44 

staff 

54 

54 

Otlier/Un known 

2 

2 


Employment Status 

Full-time 88 83 

Part-time 12 17 


(Send er 


Female 

73 

63 

Male 

27 

37 


Ra ce/Ethnicity 


Hi sp anic 

1,9 

1,3 

Asian 

1.9 

0.6 

Black 

0.5 

0.6 

Native Hawaiian or other 
Pacific Islander 

0.0 

1.3 

White 

94.7 

96.2 

Two or more 

0.5 

0.0 

Unknown 

0.5 

0.0 


Analysis and Results 

In the first phase of analysis for this research, tests were conducted to explore 
differences between mean scores by item. Also, factor analysis was performed on the data and 
no resulting constructs were observed. However, artificial "theme constructs" were created on 
an a priori basis by the researchers by assembling questions into three theme areas: Mission 
Perception, Identity Perception and Programs. The intent was to allow for more robust analysis 
by using "theme construct" means that had been created from the aggregated results from the 
several items comprising each of the Mission Perception (Table 1), Identity Perception, and 
Programs theme areas. Mean scores for theme constructs by institution and position were also 
produced (Table lb). 
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Table 1: "Mission perception" means by position and institution 


Item 

Faculty 

Staff 

All 

A 

B 

A 

B 

A 

B 

1 believe that (institution)... 







-is innovative. 

5.14 

4.97 

5.30 

5.11 

5.23 

5.04 

- is inclusive. 

5.81 

5.24 

6.10 

5.22 

5.96 

5.23 

- takes risks that are consistent with its mission and values 

5.17 

4.63 

5.41 

4.85 

5.32 

4.75 

- is pragmatic, grounding education in the realities of everyday life. 

5.67 

5.43 

5.68 

5.40 

5.69 

5.44 

-mission and values are visible to all. 

5.90 

5.47 

5.84 

5.45 

5.89 

5.48 

1 support... 







-our current approach to fostering leadership potential of individuals. 

5.51 

5.21 

6.00 

5.16 

5.80 

5.19 


Table lb: Comparison of "Mission Perception" means by institution 


A 

(n=187) 

B 

(n=145) 

All 

(n=332) 

5.66 

5.19 

5.45* 


* p <.01 


In a second phase of analysis for this research, tests were performed to investigate 
differences between subject groups using an ANOVA statistic. According to Ploutz-Snyder 
(2005), "one can apply the ANOVA statistic to answer fairly complex questions about 
differences between and within groups, overtime and across multiple conditions." 

Mean scores were determined for each DMV item for all responses, as well as by 
institution (Table 2), institution by gender, and institution by gender and position (faculty 
versus staff). The ANOVA test was performed using DMV item mean scores as dependent 
variables and "institution," "gender" and "position" as independent variables. The data indicate 
that faculty and staff share strong agreement that their institutions are inclusive and foster 
leadership potential. Additionally, faculty and staff strongly agree that the heritage of their 
institutions is evident and that a variety of services from campus ministries are offered. 
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Significant differences are seen in male and female perceptions of mission and identity, and 


there are significant differences between part-time and full-time employees on perceptions of 
inclusiveness. 


Table 2: Item means by institution 


DMV Inventory Item 

A 

(N=208) 

B 

(N = 162) 

7 Atmosphere that fosters mutual understanding and respect 

5.89 

5.30 

8 Care for each community member 

6.03 

5.55 

9 Innovation and pursuit of new and effective approaches 

5.23 

5.01 

10 Inclusive: provide access for all 

5.96 

5.21 "* 

11 Risks consistent with mission and values 

5.32 

4.71 ”* 

12 Pragmatic: grounded in realities of everyday life 

5.69 

5.42 

13 Mission and values are visible to all 

5.89 

5.46 ** 

14 Religious heritage remains relevent 

6.03 

5.82 

15 Support institution's approach to expressing identity 

5.83 

5.38 "* 

16 Fostering leadership potential of individuals 

5.80 

5.17 *** 

17 Freely examine religious and secular value systems 

5.58 

5.28 * 

18 Curricula express Catholic identity 

5.76 

5.35 "* 

19 Support approach to expressing Catholic identity 

5.70 

5.51 

20 Campus Ministry provides a variety of services and programs 

6.32 

5.90 *** 

21 Demonstrates connectedness to the global community 

6.20 

5.76 


*p<.05 **p<.01 ***p<.001; DMV response mode 7= Strongly Agree l=Strongly Disagree 


The data show agreement between female and male respondents on perceptions of an 
atmosphere of respect, fostering leadership, and promoting connectedness to a global 
community. However, females perceive greater inclusiveness, caring, and availability of 
campus ministry services, while males perceive risk-taking is consistent with the mission and 
agree more strongly that the mission is visible to all. Not surprisingly, results indicate that part- 
time staff members are significantly different in their perception of inclusiveness of the 
community. 
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Table 3: Means by institution and gender 


DMV Inventory Item 

A 

B 

Female 

(N=152) 

Male 

(N=55) 

Female 

(N=100) 

Male 

(N=59) 

7 

Atmosphere that fosters mutual understanding and respect 

5.88 

5.91 

5.24 

5.38 ••• 

8 

Care for each community member 

6.08 

5.87 

5.51 

5.63 •• 

9 

Innovation and pursuit of new and effective approaches 

5.34 

4.91 

5.25 

4.59 • 

10 

Inclusive: provide access for all 

6.04 

5.70 

5.26 

5.09 ••• 

11 

Risks consistent with mission and values 

5.36 

5.21 

4.96 

4.26 ••• 

12 

Pragmatic: grounded in realities of everyday life 

5.73 

5.57 

5.62 

5.09 • 

13 

Mission and values are visible to all 

5.90 

5.85 

5.56 

5.27 • 

14 

Religious heritage remains relevant 

6.17 

5.62 

5.98 

5.55 •• 

15 

Support institution's approach to expressing identity 

5.92 

5.57 

5.59 

5.00 ••• 

16 

Fostering leadership potential of individuals 

5.86 

5.63 

5.33 

4.89 ••• 

17 

Freely examine religious and secular value systems 

5.70 

5.24 

5.38 

5.09 • 

18 

Curricula express Catholic identity 

5.90 

5.35 

5.60 

4.93 ••• 

19 

Support approach to expressing Catholic identity 

5.88 

5.19 

5.76 

5.09 ••• 

20 

Campus Ministry provides a variety of services and programs 

6.45 

5.96 

6.11 

5.54 ••• 

21 

Demonstrates connectedness to the global community 

6.29 

5.94 

5.93 

5.45 ••• 


*p<.05 **p<.01 ***p<.001; DMV response mode 7= Strongly Agree l^Strongly Disagree 


The results indicate that mean responses to DMV items differ, despite similar 
institutional characteristics, by institution and by "theme constructs." Mean differences 
observed between institutions by "theme constructs" are significant for Mission Perception 
(p<.01). Identity Perception (p<.01), and Programs (p<.00). 

There are also differences in mean perception scores by respondent position (staff 
versus faculty) for institutions overall. Gender differences are observed on mean scores within 
and between institutions (Table 3). Finally, mean scores for DMV items differ by institution by 
gender and position (Table 4). 

Discussion 

These results conform to the principles of the two-step flow model of communication. 
Based on the reimagined construction of the two-step flow model of communication (Figure 1) 
showing institution "mission and identity" as the medium of communication, it is observed in 
these results that the institution, or "opinion leader," influences perceptions of mission and 
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identity by the research subjects, here the faculty and staff, in distinct ways that result in 
observed differences between institutions, in some cases, to significant levels. 

Table 4: Item means by institution by gender and position 




A 

B 


DMV Inventory Item 

Female 

Male 

Female 

Male 


Faculty 

(N=54) 

Staff 

(N=96) 

Faculty 

(N=36) 

Staff 

(N=16) 

Faculty 

(N=39) 

Staff 

(N=59) 

Faculty 

(N=32) 

Staff 

(N=25) 

7 

Atmosphere that fosters mutual understanding and respect 

5.92 

5.84 

5.81 

6.08 

5.34 

5.16 

5.50 

5.21 • 

S 

Care for each community member 

6.20 

6.00 

5.97 

5.77 

5.31 

5.60 

5.77 

5.46 • 

9 

Innovation and pursuit of new and effective approaches 

5.36 

5.31 

4.77 

5.23 

5.06 

5.40 

4.70 

4.42 

10 

Inclusive: provide access for all 

5.90 

6.11 

5.68 

6.00 

4.91 

5.46 

5.50 

4.58 •** 

11 

Risks consistent with mission and values 

5.20 

5.43 

5.13 

5.31 

4.86 

5.02 

4.20 

4.32 " 

12 

Pragmatic: grounded in realities of everyday life 

5.80 

5.68 

5.45 

5.69 

5.77 

5.49 

4.97 

5.13 • 

13 

Mission and values are visible to all 

5.92 

5.87 

5.87 

5.62 

5.53 

5.56 

5.30 

5.17 

14 

Religious heritage remains relevant 

6.10 

6.21 

5.45 

6.00 

5.74 

6.11 

5.57 

5.50 • 

15 

Support institution's approach to expressing identity 

5.74 

6.01 

5.48 

5.85 

5.24 

5.78 

4.77 

5.26 "* 

16 

Fostering leadership potential of individuals 

5.59 

5.99 

5.37 

6.08 

5.39 

5.27 

4.93 

4.83 •** 

17 

Freely examine religious and secular value systems 

5.48 

5.80 

5.03 

5.85 

5.21 

5.47 

4.87 

5.38 • 

18 

Curricula express Catholic identity 

5.74 

5.98 

5.23 

5.62 

5.41 

5.69 

4.79 

5.04 "• 

19 

Support approach to expressing Catholic identity 

5.72 

5.95 

5.06 

5.69 

5.50 

5.89 

4.80 

5.42 

20 

Campus Ministry provides a variety of services and programs 

6.40 

6.47 

5.68 

6.38 

5.82 

6.28 

5.50 

5.58 "• 

21 

Demonstrates connectedness to the global community 

6.36 

6.24 

5.71 

6.38 

5.71 

6.05 

5.47 

5.42 "• 


*p<.05 **p<.01 ***p<.001; DMV response mode 7= Strongly Agree l^Strongly Disagree 


With regard to Engagement Theory (Kuh et al., 1997), differences observed in the 
results of this research by institution on the Programs theme construct concur with findings 
from many years of research using the National Survey of Student Engagement (NSSE, 
http://nsse.iub.edu/html/reports.cfm) . This research suggests that campus programs, as 
described in the language of the DMV items, do have an effect on respondent perceptions of 
institutional initiatives and that they can differ by institution; however, caution is advised is 
interpreting the results due to the research having been limited to two institutions. 

Further research in the domain of qualitative methodology is suggested as a means to 
investigate the cause of disparity in results by gender. Also, content analysis can be utilized to 
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explore the relative differences in emphasis on individual constructs in the texts of the 
institutions' mission statements and linked conceptually to the quantitative results obtained 
here so deeper understanding of the dynamics at work at each place can be understood. The 
quantitative results can be used to assess how strongly faculty and staff perceive the 
communication of mission on campus and which areas need to be strengthened. Comparison of 
these results can also be made with available data on student perceptions from engagement 
studies like the National Survey of Student Engagement (NSSE) and the Higher Education 
Research Cooperative (HERI) College Student Beliefs and Values study conducted in 2004, to 
observe how employees groups compared to student groups on their perception of constructs. 
Another compelling investigation would be a comparison of results by institutions in different 
regions of the country, since data from both eastern and mid-western schools is available. 
Results can also inform policy and programming recommendations made to leaders of member 
schools by the Catholic Higher Education Research Cooperative (CHERC) and the Association of 
Catholic Colleges and Universities (ACCU). 

A worthy endeavor of the leadership of any college or university is to affirm in context 
and programs the presence of a strong and current mission statement and sense of identity. 
Responding actively to outcomes of research on campus perceptions of the environment will 
foster stronger affiliation with mission and identity and bring together the campus community 
in fruitful action that creates a rich and positive place of work and learning for all. 

Author Biography: Ellen Boylan, Ph.D., is Director of Planning and Institutional Research at 
Marywood University, Scranton, Pennsylvania. Kim Pavlick, Ph.D., is Assistant Professor of 
Communication at the University of Scranton, Scranton, Pennsylvania. 
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APPENDIX A 

DePaul Mission and Values (DMV) Inventory 
Institution Identity Scale 

Please rate the items below according to this scale: l=strongly disagree to 7=strongly agree 

1. I believe that at our very diverse personal values and religious beliefs contribute to 

an atmosphere that fosters mutual understanding and respect. 

2. I believe that we manifest personalism by our care for each member of the university 
community. 

3. I believe that University is innovative. We are never content with maintaining a 

"business as usual" approach. Our efforts are marked by innovation and single-minded 
pursuit of new and effective approaches to meet the needs of our students, society and the 
educational marketplace. 

4. I believe that University is inclusive. We provide access for all to higher education 

regardless of class, race, religion, sexual orientation, disability, ethnicity or economic 
barriers. The university community is welcoming and draws great strength from its 
diversities. 

5. I believe that University takes risks that are consistent with its mission and values. 

Historically the university has always stepped outside of tradition and beyond "status quo" 
approaches, encouraging and demonstrating an adventurous and entrepreneurial spirit. The 
measure of our success has always been the measure of our risks. 

6. I believe that University is pragmatic grounding its education in the realities of 

everyday life. Through its curricula and through the delivery of its programs and services, 
the university offers students practical solutions to their needs for higher education, career 
advancement and personal growth. 

7. I believe that University's mission and values are visible to all. Its education and 

operations are grounded in values of service, respect, justice, holistic education and 
creating quality educational opportunities especially for the underserved and disadvantage 
in our society. 

8. I believe that our religious heritage remains relevant to the university today. 

9. I support our current approach to expressing its identity. 

10. 1 support our current approach to expressing its fostering the leadership potential of 
individuals. 

11. 1 believe that our university invites all inquirers to freely examine Catholicism, other faith 
traditions and other secular values systems in light of their respective contributions to the 
human enterprise. 

12. 1 believe that the curricula at our schools and colleges have appropriate expressions of the 
university's Catholic identity. 

13. 1 support my institution's current approach to expressing its Catholic identity. 

14. Campus Ministry provides a variety of services and programs designed to serve the 
University community and enhance the institution's Catholic identity. 

15. The University sponsors a variety services and programs to demonstrate the connectedness 
to the global community. 
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APPENDIX B 

Demographic Items 


Faculty and Staff Perception of Institutional Mission and Identity 


1. What is your status? 

n FacUly 
r stair 

2. What is your current employment status? 

r* FUI time 
Partnme 

3. What is your gender? 

<■ Female 
f Male 

4. Are you Hispanic or Latino? 

r* Yes 
r M) 

5. Please indicate the racial/ethnic group with which you identify (select one or more). 

^ Amencan mdan/Maeka Name 
(” Aaan 

n Bact or Atncan Amencan 

r~ Mauve Haeialian or outer PaOSc Islander 

r witne 

*6. Please identify the institution that employs you. 
r* MnuKiod unverslty 

^ Tne unverslty or Scranton 
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Discipline and market are two related factors that are frequently used in the evaluation of 
faculty salary equity at colleges and universities (Balzer, et ah, 1996; Haignere, 2002; Luna, 
2007; Moore, 1992). Although it is assumed that the variation of market factors should be 
responsible for explaining differences in faculty salaries, this assumption has not been tested yet 
(Bellas, 1994). In general, salary differences across disciplines are considered as market neutral 
by AAUP and CUPA (Bellas, 1997; Haignere, 2002). The observed differences in faculty 
salaries at large are not considered a result of gender discrimination, but rather the effect of 
market factors; specifically, the supply of qualified faculty relative to the demand for their work 
and service by employers (Bellas, 1997; Semelroth, 1978; Waldauer, 1984). 

Eor decades, market factors have had great impact on differentiation of faculty salaries 
across disciplines at colleges and universities. Since the 1980s, differences in faculty salaries 
across disciplines have increased, especially in disciplines related to Business/Economics and 
Engineering. According to the Annual Report on the Economic Status of the Profession 
published by AAUP (1980-2010), using assistant professors and the discipline of English as 
reference groups, the proportions of average salaries in Education, Line Arts, Eoreign Languages, 
Communications, and Philosophy have decreased since 1980-81 (Eigure 1). However, the 


proportion of average salaries in Business and Economics related programs has increased from 
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124% in 1980-81 to 187% in 2009-10, representing a 51 percent increase over the time. As a 
result, faculty who have taught in fields with lower market values have suffered “pay penalty.” 
These observed differences cannot be totally explained by variances in individual characteristics 
(e.g., gender, race, and highest degree), professional maturity (e.g., rank and years of experience), 
and performance. 


% of Salani' Increase for Assistant Professor by Discipline 
1980-81 s^. :009-10 



Figure 1: Percentage of Salary Change by Discipline (1980-81 vs. 2009-10) 

Bellas (1994) points out that in higher education, the unequal job opportunities between 
male and female faculty at the entry-level position have continued, resulting in more female 
faculty concentrated in disciplines with relatively lower market values, such as English, Foreign 
Languages, Education, and Line Arts (Barbezat, 1991; Braskmp, Muffo, & Langston, 1978; 
Howard, Snyder, & McLaughlin, 1993; Raymond, Sesnowitz & Williams, 1988; Semelroth, 
1987). Consequently, the increase in differentiation of market values across disciplines has 
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further exaggerated salary differences between male and female faculty at colleges and 
universities. 

In higher education, it has become increasingly more important to develop market factors 
to appropriately explain the variability of gender differences in the study of faculty salaries 
(Luna, 2007). Haignere (2002) stressed that at a particular institution, salary differences across 
disciplines should primarily reflect the result of internal salary policies, structure, and promotion 
aiming to reward faculty based on the quality and quantity of teaching, research, and service. 

Many studies have used discipline/market factors to detect gender differences in faculty 
salaries. Some have used internal discipline/market factors to represent salary differences across 
disciplines (Braskamp & Johnson, 1978; Haignere, 2002; Reagan & Maynard, 1974). Others 
have contended that external market factors do affect departmental salary differences (Balias, 
1997; Bereman & Scott, 1991; Braskamp, Muffo & Langston, 1978; Raymond, Sesnowitz, & 
Williams, 1988). These studies have used the average salaries of newly hired assistant professors 
or disciplinary ratios obtained from national salary databases (e.g., AAUP or CUPA) to reflect 
pricing mechanism and competition at a specific discipline across institutions nationwide 
(Duncan, Krall, Maxcy, & Prus, 2004; Luna, 2007). However, the study conducted by Braskamp 
& Johnson concluded that the external market factors did affect salary differences across 
disciplines, but the relationship was not high when compared to the importance of internal 
market factors (1978). 

In studying faculty salaries at a particular institution, it seems preferable to use internal 
market factors to represent salary differences across disciplines because only internal market 
factors can fully reflect the local salary structure within that institution (Haignere, 2002; Koch & 
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Chrizmar, 1996). The more accurate the market factors reflect the salary structure and practice at 
the institution, the more reliable is the measure of market influence on faculty salaries (Reagan & 
Maynard, 1974). 

Regression Model 

Many institutional researchers have used different approaches to code disciplinary 
variables in regression, which were used to determine whether gender differences in faculty 
salaries can be appropriately explained by market/disciplinary factors after controlling other 
predictor variables. However, there is still a debate in higher education on methods of coding 
market/disciplinary variables in regression analysis. Two commonly used approaches are the 
dummy model and the market model. 

Dummy model. The dummy model is the most commonly used method, which creates a 
set of dummy disciplinary variables in order to explain salary differences across disciplines. The 
dummy model is represented by L. Haignere (2002) in Paychecks: A Guide to Conducting 
Salary-Equity Studies for Higher Education E acuity and strongly recommended by the American 
Association of University Professors (AAUP). In the dummy model, disciplinary variables are 
coded as dummy variables with one discipline serving as a default group. The goal is to 
minimize the number of disciplinary variables and maximize the statistical power in regression. 
Because faculty salaries at the entry level have fluctuated with changing market conditions and 
often resulted in a wide variation across disciplines, this approach may yield more reliable results 
(Baker, Gibbs, & Holmstrom, 1994; Reagan & Maynard, 1974). 

The dummy model is the most conservative method to detect gender differences in 
faculty salaries (Yao, 2012). Haignere (2002) summarizes that this approach allows the 
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regression to assign an appropriate value for each discipline based on faculty salaries paid in that 
discipline. The results can truly reflect the institution’s salary rewarding structure and promotion 
in practice. However, the dummy approach should be used with caution because it produces a 
large number of degrees of freedom which may limit the statistical power, particularly in a 
medium and small-size institution (Luna, 2007). More important, with a large number of dummy 
variables used in regression, it may become more complicated to explain the statistical results to 
administrators and faculty who have less knowledge and experience in multivariate statistics 
(Yao, 2012). 

Furthermore, Moore (1992) indicates that the dummy approach may not be proper to test 
gender differences in pay when a department has a very small number of faculty, or faculty in a 
department are not evenly distributed by gender. In practice, if a department has less than five 
faculty members, those faculty need to be grouped with another related discipline (Haignere, 
2002). Moreover, since different disciplines tend to have different reward structures, using the 
dummy approach may not truly reflect the internal salary rewarding structure (Howard, Snyder, 

& McLaughlin, 1993). 

Market model. Instead of using categorical variables to represent salary differences across 
disciplines, some studies transform categorical discipline variables into continuous variables or 
numerical ratios to reflect market influence on differentiation in faculty salaries. This approach 
attempts to explain gender differences in pay by assigning a market value or market ratio to each 
discipline and compare them using regression analysis. 

The market ratio is defined as a ratio of the average salary for a specific discipline 
(numerator) divided by the average salary of all disciplines combined (denominator). Luna (2007) 
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explains that the market ratio measures the relative strength of salaries between a particular 
discipline and disciplines as a whole. A market ratio below 1.0 indicates that the average salary 
in that discipline is being paid below the average salary of all disciplines combined. Conversely, 
a market ratio above 1.0 means that the average salary in that discipline is being paid above the 
average salary of all disciplines combined. In practice, market ratios that fluctuate from .95 to 
1.05 are considered in the normal range. 

The market approach has gained wide acceptance in higher education because of its 
flexibility and its convenience. Using market ratios generated from the CUPA national salary 
database in a study of faculty salaries, Luna (2007) concluded that the market ratio was the 
largest contributor to explain the variance of faculty salaries. This approach is more effective and 
efficient than the dummy model with less political and technical confusion (Luna, 2007). 
However, for a relatively small institution, using external market ratios to represent internal 
disciplines may produce a totally different salary rewarding structure, which would mask gender 
differences in pay. In addition, because gender is unevenly distributed across disciplines with 
more women are concentrated in fields with lower market values, the market ratio itself may be 
involved with gender discrimination in pay. 

In summary, to select market ratios, some researchers prefer to use internal market values 
to replace dummy coded discipline variables in faculty salaries. They argue that salary 
differences should primarily reflect the result of internal salary policies, structure, and promotion. 
Others support using external market values to truly reflect the economic competition in a 
particular discipline across institutions of higher education. In this study, two different market 
ratios, including internal market ratios and external market ratios, were created to test which type 
of market ratios would be the better one to detect gender differences in faculty salaries. 
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Methods 

Data were obtained from a unionized, four- year public institution in the northeastern 
region. The sample included 248 full-time faculty members. Among them, the distribution of 
faculty in current rank consisted of 13.7% full professors, 32.7% associate professors, 43.9% 
assistant professors, and 9.7% lecturers. The distribution of male and female faculty was 60.5% 
and 39.5%, respectively. Moreover, 18.1% were minority faculty. 

In this study, three regression models (i.e., dummy model, external market model, and 
internal market model) were developed using three different types of disciplinary variables 
(dummy variables, internal market ratios, & external market ratios) in regression. The dummy 
model assigned a set of 19 disciplines as dummy variables, with one serving as the default group. 
The external market model converted 20 disciplinary variables into numerical ratios using the 
CUPA average salaries represented in each related discipline. The internal market model 
transformed 20 disciplinary variables into numerical ratios using the average salaries of each 
discipline at the given institution. Finally, multiple regression analyses were applied to test 
which model was the best one to properly explain gender differences in faculty salaries. 

Variables used for assessing faculty salaries were selected based on strong determination 
on faculty salary rewards in the literature review and availability of data in the HR salary 
database at the given institution. Haignere (2002) warned that predictor variables should be 
carefully selected and evaluated. Some variables may be potentially tainted variables (e.g., rank 
& tenure status) which would mask gender difference in pay; some may produce redundant 
information (e.g., years in current rank and years of service at the institution) because the 
curvilinearity may occur and affect the time-related variable, particularly at unionized 
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institutions (Haignere, 2002). In this study, the categorical regression analysis was applied to test 
whether assignment of current rank was biased. In addition, the quadratic term was used for the 
variable of years of service in order to solve the curvilinear issue. 

The dependent variable was 9-10 month base salaries reported in October 2010 at the 
given institution. The predictor variables include the total number of years of service at the given 
institution, gender, race/ethnicity, highest degree earned, current rank, discipline, and market 
ratio. Disciplinary variables are coded into three different types (i.e., k- \ dummy variable, 
external market ratio, and internal market ratio). 

Total number of years of service. This variable measures the professional maturity of an 
individual faculty member who has worked toward her/his profession. It assumes that the longer 
an individual faculty member has worked in her/his professional field, the higher she/he should 
be paid. To solve the issue of curvilinearity, the quadratic term was created and used in 
regression. The result showed that the quadratic term of the total number of years of service did 
not significantly contribute to the regression model and its unstandardized coefficient was very 
small; thus, it was deleted from the final model. 

Gender and race/ethnicity. It should be realized that many published studies include 
gender but exclude race/ethnicity because there are not a sufficient number of minority faculty to 
reliably estimate the salary differences using statistical techniques in regression analysis 
(Barbezat, 2002). But ignoring race/ethnicity may mask the gender differences in pay (Haignere, 
2002). These two variables were coded as dummy variables, with male and white faculty serving 
as the reference groups. 

Current rank. Current rank is a strong determinant that reflects the institutional 
recognition of an individual faculty member’s performance based on teaching, research, and 
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service. Current ranks were categorized into four subgroups: full professors, associate professors, 
assistant professors, and lecturers and further coded as dummy variables, with the group of 
assistant professors serving as the reference group. 

Some researchers are concerned that gender may be potentially tainted in assigning the 
current rank (Allard, 1984; Barbezat, 1991; Becker & Toutkoushian, 2003; Haignere & 
Eisenberg, 2002; Scott, 1977; Smart, 1991). To test whether assignment in the current rank was 
significantly different between male and female faculty, a categorical modeling analysis (i.e., 
multinomial modeling) was conducted. The result indicated that there was a potential bias when 
assigning female faculty from assistant professors to associate professors, with the odds ratio of 
1.95. Even though the odds ratio was not statistically significant, it should be realized that using 
the current rank as a predictor variable may underestimate gender differences in pay. 

Highest degree earned. This variable represents an individual’s career investment in 
her/his profession. One dummy variable was coded, with the group of Ph.D. serving as the 
default group. In addition. Master of Eine Arts (MEA) was considered a terminal degree. 

Discipline. This variable is used to reflect the market influence on differences in faculty 
salaries. It assumes that pay differences should be associated with different market values in 
disciplines, but should not attribute to either gender or race bias (Haignere & Ein, 2002). In this 
study, twenty disciplines were transformed into three different types of disciplinary variables, 
such as k- \ dummy variables, internal market ratios, and external market ratios. 

Market ratio. This variable measures how well faculty at a particular discipline should be 
paid compared to all disciplines combined. The purpose is to reduce a large number of degrees of 
freedom and increase the statistical power in regression analysis. The formula is listed below: 
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Average salary at a particular discipline 
Market ratio = 


Average salary of all disciplines combined 


Table 1 


Disciplinary Ranking Based on Market Ratios 



Internal Market Ratios 

External Market Ratios 

Ratio 

Ranking 

Department 

Department 

Ranking 

CUPA Ratio 

1.42 

1 

Business & Economics 

Business & Economics 

1 

1.34 

1.07 

2 

Mathematics 

Biology 

2 

1.05 

1.05 

3 

Chemistry 

Sociology 

3 

1.02 

1.02 

4 

Philosophy 

Geography 

4 

1.02 

1.02 

5 

Psychology 

Anthropology 

5 

1.02 

1.01 

6 

Education 

Political Science 

6 

1.02 

1.00 

7 

Human Ecology 

Chemistry 

7 

1.01 

0.99 

8 

Sociology 

Physics 

8 

1.01 

0.99 

9 

Geography 

Earth Science 

9 

1.01 

0.98 

10 

Biology 

Psychology 

10 

1.00 

0.97 

11 

Anthropology 

Philosophy 

11 

0.98 

0.97 

12 

History 

Education 

12 

0.98 

0.97 

13 

English 

Communication 

13 

0.98 

0.96 

14 

Music 

Mathematics 

14 

0.97 

0.95 

15 

Communications 

History 

15 

0.97 

0.95 

16 

Fine Arts 

Human Ecology 

16 

0.94 

0.95 

17 

Physics 

Music 

17 

0.93 

0.93 

18 

Earth Science 

Fine Arts 

18 

0.93 

0.92 

19 

Political Science 

English 

19 

0.90 

0.87 

20 

Foreign Languages 

Foreign Language 

20 

0.92 


Note. 


1. A disciplinary ratio was calculated by using the average salary of a specific discipline divided by 
the average salary of all discipline combined at the given institution. 

2. CUPA ratios were generated based on data obtained from College and University Professional 
Association for Human Resources (CUPA-HR, 2010). 


In this study, two types of market ratios were developed (i.e., internal market ratios and 
external market ratios). The internal market ratio was calculated based on the average salaries of 
each discipline generated within the local institution. It reflects the local salary rewarding 
policies, structure, and promotion. The external market ratio was calculated based on the national 


average salaries by discipline published by CUPA in 2010. It takes into account the national 
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comparability and competition of salaries at a particular discipline across institutions of higher 
education. Table 1 showed the discipline rankings based on market ratios generated from the 
local institution and the CUPA salary database. 

Three research questions were proposed: (1) which model would be the best fit in terms 
of adjusted and F-ratio, (2) which type of disciplinary variables (i.e., k-\ dummy, internal 
market ratio, external market ratio) would be better than the other two to properly explain gender 
differences in faculty salaries based on examining the unstandardized coefficient (B) and t-value 
of the gender variable, and (3) which type of market ratios largely contribute to explain salary 
differences. 


Limitations 

Even though the variable of current rank is one of the most accessible proxies for 
measuring professional maturity and productivities, omission of variables related to measuring 
faculty performances in teaching and research would affect the strength of explanation in 
regression analysis (Moore, 1992; Webster, 1995). In addition, three disciplines were removed 
because the numbers of faculty in each discipline were less than five. For fulfilling statistical 
requirements, these faculty members were grouped into other related disciplines. 

Results 

Three multiple regression models (i.e., dummy model, internal market model, & external 
market model) were developed and compared. The purpose is to test which model is the best to 
appropriately explain the gender differences in faculty salaries at the given institution. First, the 
adjusted R and F-value generated from the three models were used to examine which model 
would be the best fit. Second, the unstandardized coefficients (B) and Lvalues of female were 
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compared to see which model would be the best to properly detect gender differences in faculty 
salaries. In addition, to test which type of market ratios (i.e., internal market ratio vs. external 
market ratios) would be better to explain faculty salaries, standard errors, t-values and partial 
correlations generated from the three regression models were compared. 

To avoid multicollinearity, values of variance inflation factors (VIFs), tolerance, and 
condition index produced by the three models were carefully examined. The issue of 
multicollinearity was not found. 


Model Fit 

The adjusted R and F-value generated from the three models are used to compare which 
model would be the best than the other two to explain differences in faculty salaries. The 
percentage of adjusted is used to test the loss of predictive power. It measures how much 
variances in faculty salaries could be accounted for by the predictor variables. The F-ratio is used 
to assess the overall fit of the regression model (Field, 2009). 

Table 2 


Summary of the Three Regression Models (N = 248) 


Model 

R 

R Square 

Adjusted R 
Square 

F 

Sig.F 

Internal Market Model 

.845 

.715 

.705 

74.83 

.000 

External Market Model 

.817 

.667 

.656 

59.79 

.000 

Dummy Model 

.852 

.727 

.694 

22.58 

.000 


Note. Significant Levels: *** p < .001, ** p < .01, * p < .05. 


As shown in Table 2, it is clear that the internal market model produced the largest 
adjusted R^ and the highest F-ratio, with adjusted R^ =.705 and F(S, 239) = 74.83 (p < .001). The 
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results provided strong evidence that the internal market model was better than the other two 
(i.e., dummy model and external market model) to explain the differences in faculty salaries. 

Gender Dijferences in Faculty Salaries 

To examine which model would be the best to properly detect gender differences in 
faculty salaries, the unstandardized coefficients (5) and t- values of female faculty were 
examined and compared among the three models. 


Table 3 

Unstandardized Coejficients and t-values of Female Faculty Generated from the Three 
Regression Models (N = 248) 



Unstandardized 

Coefficients 

Standardized 

Coefficients 




B 

P 

t 

Sig. 

Dummy Model 

-470.0 

-.023 

-.59 

.554 

Internal Market Model 

-364.9 

-.015 

-.41 

.680 

External Market Model 

461.6 

.018 

.48 

.631 


Note. Significant Levels: *** p < .001, ** p < .01, * p < .05. 


As shown in Table 3, although t-values of female faculty generated from the three models 
were very small and not statistically significant, it was apparent that the unstandardized 
coefficients of female faculty produced by the three models largely varied, ranging from -$470 to 
$462. The results showed that the unstandardized coefficient of female faculty produced by the 
internal market model was much closer to the one generated from the dummy model. 

Market Ratios 

To test which type of market ratios was better to explain faculty salaries using internal 
and external market ratios, standard errors, t-values and partial correlations were compared. As 




Faculty Salary Study 13 


shown in Table 4, the internal market ratios obtained a higher t- value (t = 16.14, p < .001) but a 
lower standardized error ($3,5 14) compared to the external market ratios. In addition, the partial 
correlation for the internal market ratios (partial r = .722) was much higher than the one 
associated with the external market ratios (partial r = .664), indicating that 72.2% of accounted 
variances in faculty salaries could be explained by the internal market ratios. As a result, the 
internal market ratio was the better one to explain differences in faculty salaries at the given 
institution. 

Table 4 


Unstandardized Coefficients of Market Ratios Generated from Internal and External 
Market Models (N = 248) 



Unstandardized Coefficients 



Correlation 

Discipline/Market Variable 

B 

Std. Error 

t 

Sig. 

Partial 

Internal Market Ratios 

56722.56 

3514.00 

16.14 

.000 

.722 

External Market Ratios 

61870.01 

4502.68 

13.74 

.000 

.664 


Note. Significant levels: *** p < .001, ** p < .01, * p < .05. 


Conclusions 

Salary differences across disciplines have been growing for decades at colleges and 
universities. Along with the high percentage of female faculty concentrated in disciplines with 
lower market values, gender equity continues to be an important issue in the study of faculty 
salaries. Previous studies of salary-equity have often examined relatively large datasets; however, 
these results cannot directly address conditions at individual institutions (Toutkoushian, 2002). 
Ferber and Loeb (2002) argued that even if gender differences in pay exist in higher education, 
the pay gap may not happen at a particular institution. Therefore, there is a continued need for 


studies on the issue of gender-equity in faculty salaries at institutional levels. 
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This study supports the premise that a single, continuous variable can be used to replace 
categorical discipline variables to explain variances in faculty salaries at a small-size public 
institution. Many studies have examined salary equity using relatively large datasets which allow 
applying sufficient degrees of freedom in multiple regression analyses (Haignere, 2002; Luna, 
2007). However, for those relatively small institutions with few women and minority faculty 
members, it is difficult to run multiple regression analysis incorporated with large numbers of 
dummy coded disciplinary variables (Ferber & Loeb, 2002; Toutkoushian, 2002). Using market 
ratios to replace dummy coded disciplinary variables can properly solve this issue. 

Secondly, this study demonstrates that the internal market ratio may serve as a better 
indicator to represent disciplinary differences in testing gender differences in faculty salaries 
because it truly reflects the local institution’s salary rewarding structure (Balzer et ah, 1996; 
Braskamp, Muffo, & Langson, 1978; Reagan & Maynard, 1974; Raymond, Sesnowitz, & 
Williams, 1988). 

Internal market ratios and external market ratios are highly related with each other but 
apparently reflect different salary rewarding structures. As shown in Table 1, it is clear that the 
salary structure developed based on the internal market ratios is ranked differently from the one 
created by the external market ratios. For example, the discipline of Mathematics ranks top 
second in salary rewarding structure at the given institution, with an internal market ratio of 1.07; 
while the national CUPA ratios showed that salary rewarding to faculty in Mathematics ranks 14 
out of 20, with a market ratio as low as 0.97. Even the discipline of Business and Economics, 
which is ranked as the top paying discipline in both internal and external salary structures, the 
values of the market ratios appear to be different (1.42 vs. 1.34). Therefore, inclusion of external 
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market ratios in salary studies should be carefully examined before uploading into the regression 
model. 

Thirdly, there are no universal guidelines that can be used to determine how to conduct 
salary-equity studies. Although the AAUP has provided primary guidelines in Paychecks 
(Haignere, 2002) based on twelve case studies in the SUNY system, the use of dummy coded 
disciplinary variables to represent salary differences across disciplines have been viewed with 
skepticism. This approach may result in larger measurements of salary inequity (Toutkoushian, 
2002). In this study, the unstandardized coefficients of female faculty generated by the dummy 
model and external market model were largely different (-$470 vs. $ 462), resulting in $932 
difference in an absolute value. The same issue also occurred in Luna’s study using a large 
dataset collected from multiple institutions (2007, pp. 7-8). The results may lead to a 
controversial conclusion that the external market approach should be used with caution 
compared to using the internal market model at an individual institution. 

In conclusion, market factors become major sources for testing gender differences in 
faculty salaries. Because female faculty tend to be concentrated in disciplines with lower market 
values, the market ratio itself may incorporate with gender discrimination (Bellas, 1997; Moore, 
1992). Thus, inclusion of market ratios in examining faculty salaries may mask the potential 
gender discrimination in salary rewards. If a college is a leading institution with relatively 
competitive salaries paid to faculty, using the external market ratios may truly represent its 
mission and goals of recruiting and rewarding the best faculty. However, for most medium and 
small-size institutions with less competitive salaries rewarded to faculty, it is evident that the 
internal market ratios may be better than the external market ratios in predicting gender 


differences in faculty salaries. 
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